xlang: the snake_case conversion of field names is not specified, and the runtimes do not agree

Aperta
  • Qualcosa non va?
#4,187 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@dhruv-dagar ci sta già lavorando.

Dal 19/10/2026.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp, csharp, dart, go, java, javascript, kotlin, python, rust, scala, swift
Ambito
backend, distributed-systems

Direzione di ricerca

Start with docs/specification/xlang_serialization_spec.md at the cited lines, then compare the conversion functions and cross-language tests named in the issue. The proposal requires choosing and specifying one exact conversion, applying it consistently to written names, sort order, and fingerprints across runtimes, and adding known-answer cases. Done also requires documenting how existing data with affected field names is handled.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug
Search before asking
  • I had searched in the issues and found no similar issues.
Version

All

Component(s)

Other

Minimal reproduce step

Problem

The xlang specification says that field names are "converted to snake_case" (docs/specification/xlang_serialization_spec.md, lines 882 and 1732), and line 1800 refers to "the snake_case field name". It does not define the conversion. Each runtime has its own conversion, and the results are different for acronyms, an acronym followed by a digit, a leading capital letter, and non-ASCII capital letters. Rust and Python write the declared field name without a conversion.

Java, Kotlin, Scala, JS, Dart, C#, Swift, and C++ agree for lowerCamelCase names that have only single ASCII capital letters: they all write createdAt as created_at. A Go field must be exported, so Go writes created_at for CreatedAt. Rust and Python write the declared name, so they match only for a field declared as created_at.

Field name written to TypeDef/TypeMeta (apache/main at a11d257b4)

Every cell comes from a probe program that ran the runtime code at a11d257b4 and, where possible, encoded and decoded a real TypeDef/TypeMeta.

Input Java, Kotlin, Scala, JS Go Dart C#, Swift C++ Rust Python
createdAt created_at (1) created_at created_at created_at createdAt (2) createdAt
fooURL foo_u_r_l (1) foo_u_r_l foo_url foo_url fooURL (2) fooURL
URLValue _u_r_l_value u_r_l_value u_r_l_value url_value url_value URLValue (2) URLValue
HTTP2Server _h_t_t_p2_server h_t_t_p2_server h_t_t_p2_server htt_p2_server http2_server HTTP2Server (2) HTTP2Server
CreatedAt _created_at created_at created_at created_at created_at CreatedAt (2) CreatedAt
x2Y x2_y (1) x2_y x2_y x2_y x2Y (2) x2Y
fooÖl fooÖl (1) (3) foo_öl fooÖl (4) (4)
already_snake already_snake (1) already_snake already_snake already_snake already_snake already_snake
  1. Go writes only exported fields, so a name that starts with a lowercase letter does not reach a Go TypeDef. The SnakeCase helper gives created_at, foo_u_r_l, x2_y, and fooÖl for these names.
  2. The Rust runtime writes the raw name (probe of TypeMeta::new with FieldInfo::new), but #[derive(ForyStruct)] does not compile for a field name with an ASCII capital letter (error E0063, "missing field"). The derive converts identifiers to snake_case at object/util.rs:410 and :422.
  3. fooÖl is not a valid Dart identifier.
  4. Rust and Python stop with "Unsupported character for LOWER_UPPER_DIGIT_SPECIAL encoding". Java writes the name as UTF-8.

Some runtimes use a second conversion for the field sort order and the struct fingerprint, and for some inputs it gives a different result from their written name:

Input JS sort and fingerprint Rust sort and fingerprint Python sort and fingerprint
fooURL foo_url foo_url foo_url
URLValue url_value url_value url_value
HTTP2Server htt_p2_server htt_p2_server http2_server
CreatedAt created_at created_at created_at
fooÖl fooÖl fooÖl foo_öl

In C#, Swift, and C++, the second conversion gave the same result as the written name for every input.

Conversion functions

  • Java: StringUtils.lowerCamelToLowerUnderscore (StringUtils.java:259), applied at TypeDefEncoder.java:277. The probe built an xlang TypeDef, wrote each field with TypeDefEncoder.writeFieldsInfo, and found the decoded names in TypeDef.getEncoded(). Kotlin and Scala use this fory-core path and were not probed separately.
  • JavaScript: lowerCamelToLowerUnderscore (TypeMeta.ts:979) writes the name (TypeMeta.ts:917). toSnakeCase (TypeMeta.ts:1059) gives the sort key, the fingerprint, and the read match. The probe encoded TypeMeta.fromTypeInfo(...).toBytes() and decoded the names.
  • Go: SnakeCase (go/fory/util.go:60), applied at field_spec.go:427 and written at type_def.go:509. Unexported fields are skipped at type_def.go:494. The probe ran buildTypeDef and decodeTypeDef.
  • Dart: _toSnakeCase (fory_generator.dart:5739, called at :1984). The runtime writes the generated identifier unchanged (type_resolver.dart:1181). The probe ran build_runner and read TypeDef.encoded.
  • C#: the generator's ToSnakeCase (ForyModelGenerator.Analysis.cs:2803, called at :1490) and the runtime's LowerCamelToLowerUnderscore (TypeMeta.cs:81, applied at :333). Both use char.IsUpper, so they convert non-ASCII capital letters. The probe compiled a [ForyStruct] class with the source generator and round-tripped TypeMeta.Encode and Decode. A type with both createdAt and CreatedAt fails the build with FORY016 ("duplicate structural field identity 'created_at'").
  • Swift: lowerCamelToLowerUnderscore (TypeMeta.swift:1123, applied at :267). The probe compiled Sources/Fory and round-tripped TypeMeta.encode and decode. The macro's toSnakeCase (ForyObjectMacro.swift:3698) gives the sort order; the probe ran a verbatim copy of it, because the macro build needs swift-syntax.
  • C++: to_snake_case (cpp/fory/util/string_util.h:287), applied at type_resolver.h:1183. The probe built fory_serialization, ran FieldInfoBuilder::build(), and round-tripped TypeMeta::to_bytes and from_bytes. The sort order uses normalize_field_name (type_resolver.cc:1540), which calls a second, file-local to_snake_case (type_resolver.cc:1479).
  • Rust: the derive passes the raw identifier to FieldInfo::new_with_id (rust/fory-derive/src/object/field_codec.rs:201). The probe ran TypeMeta::new with FieldInfo::new and decoded the result. to_snake_case (string_util.rs:110) gives the sort order and the fingerprint.
  • Python: the raw dataclass attribute (meta/typedef.py:1302), written by typedef_encoder.write_field_info (typedef_encoder.py:247), which the probe ran with a recording buffer. _to_snake_case (struct.py:1183) gives the sort order and the fingerprint. The probe did not run build_field_infos or the full encode_typedef, because pyfory does not import without its native extension.

Proposal

Define one exact conversion in the xlang specification, apply it in every runtime for the written name, the sort order, and the fingerprint, and add known-answer cases (for example, the names in the table above) to the cross-language tests. A change to the conversion changes the wire names of some fields, so the specification must also say how existing data with such names is handled.

Related

We found no existing issue for this problem. The row-format schema-evolution work in #3714 uses the Java field-name conversion for its wire names and documents the portable subset, ordinary lowerCamelCase names, until this issue is resolved.

What did you expect to see?

xlang schema should always produce the same normalized names

What did you see instead?

different languages produce different names for the same schema

Anything Else?

Issue diagnosed with Claude Opus 5.5

Are you willing to submit a PR?
  • I'm willing to submit a PR!
Lingua principale
Java
Stelle
4.6k
Fork
470
Merge medio
9h 28m
PR unite (30g)
33

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/fory

Tutte le issue di apache/fory

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.