sql: casting FLOAT in [32767, 32767.5) to INT2 fails with "integer out of range" only in the vectorized engine

Open
#176,149 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, sql
Domain
databases

Research direction

Start with floatToInt in pkg/sql/colexec/execgen/cmd/execgen/cast_gen_util.go and compare it with pkg/sql/sem/eval/cast.go. Run the supplied SET vectorize queries around the INT2 and INT4 bounds, then exercise the vectorized cast implementation identified by EXPLAIN (VEC). Done means rounded values that fit the target integer width succeed consistently in both engines, with coverage for the positive and negative boundary cases.

Written by the indexing model from the issue text.

Description

C-bug O-community T-sql-queries X-blathers-triaged X-move-internal

Describe the problem

With vectorize=on, casting a FLOAT in [32767, 32767.5) or (-32768.5, -32768] to INT2 fails with integer out of range, although the rounded value is in range. With vectorize=off the cast returns 32767 / -32768. INT4 behaves the same way near its bounds.

To Reproduce

Single node, cockroach start-single-node --insecure:

CREATE TABLE t (f FLOAT8);
INSERT INTO t VALUES (32767.0), (32767.4), (-32768.4), (2147483647.2);

SET vectorize = off;
SELECT f, f::INT2 FROM t WHERE f < 40000;   -- 32767, 32767, -32768
SELECT f, f::INT4 FROM t WHERE f > 40000;   -- 2147483647
SET vectorize = on;
SELECT f, f::INT2 FROM t WHERE f < 40000;   -- ERROR: integer out of range (22003)
SELECT f, f::INT4 FROM t WHERE f > 40000;   -- ERROR: integer out of range (22003)

EXPLAIN (VEC) shows colexecbase.castFloatInt2Op.

Expected behavior

32767 / -32768 / 2147483647 in both engines (PostgreSQL also returns 32767 for 32767.4::float8::int2).

Additional context

  • Vectorized: execgen/cast_gen_util.go:263 floatToInt checks v >= float(MaxInt16) / v <= float(MinInt16) before rounding, so 32767.0 itself is rejected.
  • Row engine: sem/eval/cast.go:216 checks the INT8 range, rounds with RoundToEven, and then checks the width in AdjustValueToType.
  • Related but different: #115131 (the row engine returns 32767 for '32767.6'::float8::int2 instead of an error).

Environment:

  • CockroachDB v26.2.7 (official Docker image), Linux

Found with AI assistance.

Jira issue: CRDB-70075

Dominant language
Go
Stars
32.6k
Forks
4.2k
PR merge metrics
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from cockroachdb/cockroach

All issues in cockroachdb/cockroach

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.