Cannot access file and directory _names_ that are invalid UTF-8
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Domain
- operating-systems
Research direction
The affected entry points are the file-system functions listed in Section 9.16 of the Chez Scheme User’s Guide; the issue does not name source files or tests. Start by tracing how those functions convert names on POSIX systems and review the linked UTF-8b references. Done means invalid UTF-8 names can be accessed without loss across the listed functions.
Written by the indexing model from the issue text.
Description
The functions
(current-directory) (cd) (directory-list)
(file-exists?) (file-regular?) (file-directory?) (file-symbolic-link?) (file-access-time) (file-change-time)
(file-modification-time) (mkdir) (delete-file) (delete-directory) (rename-file) (chmod) (get-mode)
described in Section 9.16. File System Interface https://cisco.github.io/ChezScheme/csug10.0/io.html#./io:h16 operate on file names and directory names represented as Scheme strings.
When actually accessing the file system on POSIX systems, such strings are automatically converted from/to UTF-8.
This has the side effect that existing files and directories whose names are invalid UTF-8 cannot be accessed by the functions listed above.
Example:
$ mkdir example
$ cd example
$ touch $(printf 'AAA\xffzzz')
$ ls -l aaa*
-rw-r--r-- 1 user users 0 Feb 20 10:15 'AAA'$'\377''zzz'
$ chezscheme
> (define x (car (sort string<? (directory-list "."))))
> x
"AAA�zzz"
> (char->integer (string-ref x 3))
65533
> (delete-file x)
#f
> (delete-file x #t)
Exception in delete-file: failed for AAA�zzz: no such file or directory
Type (debug) to enter the debugger.
The problem is: byte #xff is not a valid UTF-8 sequence,
and (directory-list) converts it to replacement character #xFFFD as per UTF-8 error-handling rules.
As a consequence, the file created with shell command touch $(printf 'AAA\xffzzz')
and all other files or directories whose names are invalid UTF-8
cannot be accessed with Chez Scheme functions listed above.
Since POSIX file system specifications do not require that files or directory names are valid UTF-8,
this leaves the above Chez Scheme functions in the uncomfortable position of failing on some valid POSIX file and directory names.
A solution could be to convert file and directory names from/to UTF-8b (note the 'b') instead of UTF-8,
because UTF-8b is an extension of UTF-8 designed exactly to losslessly convert any byte or byte sequence.
For a definition of UTF-8b, see
https://peps.python.org/pep-0383
https://web.archive.org/web/20090830064219/http://mail.nl.linux.org/linux-utf8/2000-07/msg00040.html
- Dominant language
- Scheme
- Stars
- 7.4k
- Forks
- 1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 19
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from cisco/ChezScheme
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
cisco/ChezScheme#1110 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
cisco/ChezScheme#1111 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 12/100
cisco/ChezScheme#1101 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
cisco/ChezScheme#1078 · 1 comment ·
Maintainers usually reply within 1 day
-
RFC: solution to clangd not finding "scheme.h"Possibly taken @reginaldalfret claimed this 20 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 48/100
cisco/ChezScheme#1073 · 1 comment ·
Maintainers usually reply within 1 day
All issues in cisco/ChezScheme
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
bug high-priority
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
dividebysandwich/rust-dos#1583 ·
Maintainers usually reply within 1 day
-
bug linker sev-2
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 1/5 Under an hour Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 2 days