changefeedccl: Avro encoding of TIMESTAMP/TIMESTAMPTZ overflows `timestamp-micros` after 294247-01-10 and for `infinity`, emitting dates around year -290278
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 48/100
Direzione di ricerca
Read pkg/ccl/changefeedccl/avro/avro.go around the timestamp and date encoding cases, then review pkg/ccl/changefeedccl/avro/avro_test.go and its timestamp range TODO. Reproduce with the supplied timestamp values and Avro changefeed script. Done means out-of-range and infinite TIMESTAMP/TIMESTAMPTZ values no longer silently encode as different timestamps, with coverage for the chosen behavior.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Describe the problem
Avro changefeeds encode TIMESTAMP and TIMESTAMPTZ columns as long with logical type timestamp-micros
(microseconds since the Unix epoch). CockroachDB accepts timestamps up to 294276-12-31 23:59:59.999999, but an
int64 microsecond count only reaches 294247-01-10 04:00:54.775807. Every value after that, including
'infinity' (stored as the internal maximum), overflows silently. The message carries a large negative number,
which every Avro reader decodes as a date about 290,000 years BC. '-infinity' is sent as the internal minimum,
-4714-11-23, as if it were a real date.
No error is raised, and the changefeed keeps running. DATE columns are handled. The Avro encoder rejects an
infinite DATE with the terminal error infinite date not yet supported with avro, but TIMESTAMP and
TIMESTAMPTZ have no such check.
To Reproduce
cockroach start-single-node --insecure- Run this script (
pip install psycopg2-binary fastavro; it starts a minimal schema registry on localhost and
decodes the messages of a sinkless changefeed):
import io, json, sys, threading, psycopg2, fastavro
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
schemas = []
class Registry(BaseHTTPRequestHandler): # minimal Confluent schema registry: register -> id
def log_message(self, *a): pass
def do_POST(self):
s = json.loads(self.rfile.read(int(self.headers["Content-Length"])))["schema"]
if s not in schemas: schemas.append(s)
out = json.dumps({"id": schemas.index(s) + 1}).encode()
self.send_response(200); self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(out))); self.end_headers(); self.wfile.write(out)
srv = ThreadingHTTPServer(("127.0.0.1", 0), Registry)
threading.Thread(target=srv.serve_forever, daemon=True).start()
c = psycopg2.connect(host="127.0.0.1", port=int(sys.argv[1]) if len(sys.argv) > 1 else 26257, user="root", dbname="defaultdb")
c.autocommit = True; cur = c.cursor()
cur.execute("CREATE TABLE avts (id INT PRIMARY KEY, ts TIMESTAMP, tz TIMESTAMPTZ)")
vals = ["2024-02-29 12:00:00.000001", "294247-01-10 04:00:54.775807", "294247-01-10 04:00:54.775808",
"294276-12-31 23:59:59.999999", "infinity", "-infinity"]
for i, v in enumerate(vals):
cur.execute("INSERT INTO avts VALUES (%s, %s, %s)", (i, v, v))
cur.execute("CREATE CHANGEFEED FOR TABLE avts WITH format = avro, initial_scan = 'only', "
"confluent_schema_registry = 'http://127.0.0.1:%d'" % srv.server_address[1])
def plain(s): # drop logicalType so fastavro returns the raw long
if isinstance(s, dict): return {k: plain(v) for k, v in s.items() if k != "logicalType"}
if isinstance(s, list): return [plain(x) for x in s]
return s
for _, _, value in sorted(cur.fetchall(), key=lambda r: bytes(r[1])):
value = bytes(value) # Confluent wire format: 0x00, 4-byte schema id, Avro body
schema = fastavro.parse_schema(plain(json.loads(schemas[int.from_bytes(value[1:5], "big") - 1])))
rec = fastavro.schemaless_reader(io.BytesIO(value[5:]), schema)["after"]
days = rec["ts"] // 86_400_000_000
print(f"id={rec['id']} source {vals[rec['id']]!r:32} timestamp-micros ts={rec['ts']:>21} tz={rec['tz']:>21} "
f"(~ year {1970 + days / 365.2425:.0f})")
Output on master:
id=0 source '2024-02-29 12:00:00.000001' timestamp-micros ts= 1709208000000001 tz= 1709208000000001 (~ year 2024)
id=1 source '294247-01-10 04:00:54.775807' timestamp-micros ts= 9223372036854775807 tz= 9223372036854775807 (~ year 294247)
id=2 source '294247-01-10 04:00:54.775808' timestamp-micros ts= -9223372036854775808 tz= -9223372036854775808 (~ year -290307)
id=3 source '294276-12-31 23:59:59.999999' timestamp-micros ts= -9222426057709551617 tz= -9222426057709551617 (~ year -290277)
id=4 source 'infinity' timestamp-micros ts= -9222425971309551617 tz= -9222425971309551617 (~ year -290277)
id=5 source '-infinity' timestamp-micros ts= -210898425600000000 tz= -210898425600000000 (~ year -4713)
Decoded with the logical type, rows 2, 3 and 4 are dates in the year -290308 / -290278, e.g. infinity → -290278-12-15.
Expected behavior
A value that cannot be represented as timestamp-micros should not be emitted as a different valid timestamp.
Either the changefeed fails with a clear error, as it does for infinite DATE values, or infinity/-infinity
and out-of-range values are encoded in some lossless, documented way.
Additional data / screenshots
Cause on master: the Avro encoder returns d.(*tree.DTimestamp).Time / d.(*tree.DTimestampTZ).Time to goavro
for the timestamp-micros logical type
(avro.go#L538-L562).
Unlike the DATE case
(avro.go#L494-L498,
if !date.IsFinite() { return ... "infinite date not yet supported with avro" }), there is no infinity or range
check. goavro v2.12.0 timeStampMicrosFromNative
(logical_type.go#L215) computes
val.Unix()*1e6 + int64(val.Nanosecond()/1e3) with no overflow check, so the int64 wraps.
This was reported before as #37508 ("cdc: large dates and timestamps are silently changed", goavro overflow),
which was closed as stale in 2023 without a fix. The Avro encoder tests still restrict random timestamps to
values that fit (TODO(mjibson): goavro mishandles timestamps ...,
avro_test.go#L255-L269).
It still reproduces on master, and infinity values hit it.
Related but different: #176152, the JSON changefeed / to_json encoding of infinite timestamps as
"294277-01-01T23:59:59.999999" (out-of-range text, not a wrapped value). #32472 added the timestamp logical
types to Avro changefeeds.
Environment:
- CockroachDB version: master at d30c905fff79 (2026-09-23, current public
masterHEAD on 2026-10-18), built
from source withbazel build //pkg/cmd/cockroach-short --config=dev, and the edge binary
v26.4.0-alpha.3-dev-74a73764f18b (built 2026-10-18). Also reproduced on v26.3.2, v26.4.0-beta.2 and v25.4.17. - Server OS: Linux (Ubuntu 22.04), single node
- Client app: psycopg2 2.9.13 + fastavro 1.13.1 (sinkless changefeed with a local schema registry)
Additional context
Avro is the changefeed format used with Kafka and schema registries. A far-future sentinel such as infinity, or
a "valid until 294276" bound used as an open-ended validity period, reaches every downstream system as a date
290,000 years BC. A consumer that writes the change into another database stores that value, and an open-ended
record silently becomes one that expired before recorded history.
Jira issue: CRDB-70529
- Lingua principale
- Go
- Stelle
- 32.6k
- Fork
- 4.2k
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di cockroachdb/cockroach
-
C-bug O-community X-blathers-untriaged
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
cockroachdb/cockroach#176245 · 1 commento ·
-
sql: pg_database.dattablespace is 0 but pg_tablespace.oid is 1663, so they no longer joinForse già presa @1AbdullahButt l’ha presa 14 giorni fa. ApertaC-bug O-community X-blathers-triaged X-move-internal
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
cockroachdb/cockroach#176228 · 1 commento ·
-
PG compat: overflow in integer power calculation triggers incompatible SQLSTATEForse di nuovo libera @darshpatel1052 l’ha presa 18 giorni fa e non c’è nessuna pull request aperta. ApertaC-bug O-community X-blathers-untriaged
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
cockroachdb/cockroach#176223 · 2 commenti ·
-
util/log: bufferedSinkCloser.Close leaks a goroutine after timeoutForse già presa @dajiaohuang l’ha presa 15 giorni fa. ApertaC-bug O-community X-blathers-untriaged
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 84/100
cockroachdb/cockroach#176210 · 1 commento ·
-
C-bug O-community T-sql-queries X-blathers-triaged X-move-internal
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
cockroachdb/cockroach#176150 · 1 commento ·
Tutte le issue di cockroachdb/cockroach
Issue simili
-
☢️ Bug v3
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
I maintainer di solito rispondono entro 1 giorno
-
area/import-export area/projects concern/ux
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
go-vikunja/vikunja#4455 · 1 commento ·
I maintainer di solito rispondono entro 2 giorni
-
Dependabot PRs no longer merge automaticallyForse già presa @dwisiswant0 l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
projectdiscovery/urlfinder#160 ·
I maintainer di solito rispondono entro 1 giorno
-
database
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
chnm/legal-modernism#450 ·
I maintainer di solito rispondono entro 1 giorno
-
tech-debt
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
I maintainer di solito rispondono entro 1 giorno