openai_compatible_* functions truncate fractional temperature, top_p and penalties (chat_generate also sends them as strings)

Start with src/openai_compatible/generate.sql, where all three affected functions and the suggested changes are shown. Read the option parsing and request body construction for each function, then use the OpenAiCompletionsTest and OpenAiChatTest harnesses mentioned in the issue. Done means fractional values retain their precision and chat_generate sends them as JSON numbers.

openai_compatible_generate, openai_compatible_generate_json and openai_compatible_chat_generate don't send the temperature, top_p, presence_penalty and frequency_penalty values they're given. Fractional values are truncated, and chat_generate sends them as JSON strings.

Observed

On IBM i 7.5, the request bodies these functions sent to an OpenAI-compatible endpoint were:

So temperature 0.7 behaves like greedy decoding (0), and endpoints that validate types reject chat_generate's string value.

Cause

1. Truncation. coalesce(json_value(options, '$.temperature'), 1) mixes the VARCHAR returned by JSON_VALUE with an INTEGER, so the result is an integer:

   values coalesce(json_value('{"t":0.7}', '$.t'), 1);   -- 0
   

2. Precision. Even with that fixed, the variables are decimal(3,1), so common values like top_p 0.95 would become 0.9.
3. Strings in chat_generate. JSON_UPDATE writes a DECIMAL as a string, but writes DECFLOAT as a number:

   values json_update('{}', 'SET', '$.t', cast(0.7 as decimal(4,2)));    -- {"t":".70"}
   values json_update('{}', 'SET', '$.t', cast(0.7 as decfloat(16)));    -- {"t":0.7}
   

   (JSON_OBJECT, used by the other two functions, writes decimals as numbers.)

Fix

• Read the values with json_value(..., returning decimal(4,2)).
• Declare the four variables as decimal(4,2).
• In chat_generate, cast to decfloat(16) for JSON_UPDATE.

This was verified on IBM i 7.5 with the test harness from #47 (OpenAiCompletionsTest, OpenAiChatTest). The change to src/openai_compatible/generate.sql:

--- a/src/openai_compatible/generate.sql
+++ b/src/openai_compatible/generate.sql
@@ -60,15 +60,15 @@
   -- Extract parameters from options
   declare model_id varchar(1000) ccsid 1208;
   declare max_tokens integer default 256;
-  declare temperature decimal(3,1);
-  declare top_p decimal(3,1);
+  declare temperature decimal(4,2);
+  declare top_p decimal(4,2);
   declare n integer;
   declare stream varchar(5) ccsid 1208;
   declare logprobs integer;
   declare echo varchar(5) ccsid 1208;
   declare stop_token varchar(32000) ccsid 1208;
-  declare presence_penalty decimal(3,1);
-  declare frequency_penalty decimal(3,1);
+  declare presence_penalty decimal(4,2);
+  declare frequency_penalty decimal(4,2);
   declare logit_bias varchar(32000) ccsid 1208;
   declare user_id varchar(1000) ccsid 1208;
   declare seed integer;
@@ -77,15 +77,15 @@
   -- Get parameters from options JSON
   set model_id = json_value(options, '$.model_id');
   set max_tokens = coalesce(json_value(options, '$.max_tokens'), 256);
-  set temperature = coalesce(json_value(options, '$.temperature'), 1);
-  set top_p = coalesce(json_value(options, '$.top_p'), 1);
+  set temperature = coalesce(json_value(options, '$.temperature' returning decimal(4,2)), 1);
+  set top_p = coalesce(json_value(options, '$.top_p' returning decimal(4,2)), 1);
   set n = coalesce(json_value(options, '$.n'), 1);
   set stream = coalesce(json_value(options, '$.stream'), 'false');
   set logprobs = json_value(options, '$.logprobs');
   set echo = coalesce(json_value(options, '$.echo'), 'false');
   set stop_token = json_value(options, '$.stop');
-  set presence_penalty = coalesce(json_value(options, '$.presence_penalty'), 0);
-  set frequency_penalty = coalesce(json_value(options, '$.frequency_penalty'), 0);
+  set presence_penalty = coalesce(json_value(options, '$.presence_penalty' returning decimal(4,2)), 0);
+  set frequency_penalty = coalesce(json_value(options, '$.frequency_penalty' returning decimal(4,2)), 0);
   set logit_bias = json_value(options, '$.logit_bias');
   set user_id = json_value(options, '$.user');
   set seed = json_value(options, '$.seed');
@@ -261,15 +261,15 @@
   -- Extract parameters from options
   declare model_id varchar(1000) ccsid 1208;
   declare max_tokens integer default 256;
-  declare temperature decimal(3,1);
-  declare top_p decimal(3,1);
+  declare temperature decimal(4,2);
+  declare top_p decimal(4,2);
   declare n integer;
   declare stream varchar(5) ccsid 1208;
   declare logprobs integer;
   declare echo varchar(5) ccsid 1208;
   declare stop_token varchar(32000) ccsid 1208;
-  declare presence_penalty decimal(3,1);
-  declare frequency_penalty decimal(3,1);
+  declare presence_penalty decimal(4,2);
+  declare frequency_penalty decimal(4,2);
   declare logit_bias varchar(32000) ccsid 1208;
   declare user_id varchar(1000) ccsid 1208;
   declare seed integer;
@@ -278,15 +278,15 @@
   -- Get parameters from options JSON
   set model_id = json_value(options, '$.model_id');
   set max_tokens = coalesce(json_value(options, '$.max_tokens'), 256);
-  set temperature = coalesce(json_value(options, '$.temperature'), 1);
-  set top_p = coalesce(json_value(options, '$.top_p'), 1);
+  set temperature = coalesce(json_value(options, '$.temperature' returning decimal(4,2)), 1);
+  set top_p = coalesce(json_value(options, '$.top_p' returning decimal(4,2)), 1);
   set n = coalesce(json_value(options, '$.n'), 1);
   set stream = coalesce(json_value(options, '$.stream'), 'false');
   set logprobs = json_value(options, '$.logprobs');
   set echo = coalesce(json_value(options, '$.echo'), 'false');
   set stop_token = json_value(options, '$.stop');
-  set presence_penalty = coalesce(json_value(options, '$.presence_penalty'), 0);
-  set frequency_penalty = coalesce(json_value(options, '$.frequency_penalty'), 0);
+  set presence_penalty = coalesce(json_value(options, '$.presence_penalty' returning decimal(4,2)), 0);
+  set frequency_penalty = coalesce(json_value(options, '$.frequency_penalty' returning decimal(4,2)), 0);
   set logit_bias = json_value(options, '$.logit_bias');
   set user_id = json_value(options, '$.user');
   set seed = json_value(options, '$.seed');
@@ -450,24 +450,24 @@
   -- Extract parameters from options
   declare model_id varchar(1000) ccsid 1208;
   declare max_tokens integer default 256;
-  declare temperature decimal(3,1);
-  declare top_p decimal(3,1);
+  declare temperature decimal(4,2);
+  declare top_p decimal(4,2);
   declare n integer;
   declare stop_token varchar(32000) ccsid 1208;
-  declare presence_penalty decimal(3,1);
-  declare frequency_penalty decimal(3,1);
+  declare presence_penalty decimal(4,2);
+  declare frequency_penalty decimal(4,2);
   declare user_id varchar(1000) ccsid 1208;
   declare seed integer;
 
   -- Get parameters from options JSON
   set model_id          = json_value(options, '$.model_id');
   set max_tokens        = coalesce(json_value(options, '$.max_tokens'), 256);
-  set temperature       = coalesce(json_value(options, '$.temperature'), 1);
-  set top_p             = coalesce(json_value(options, '$.top_p'), 1);
+  set temperature       = coalesce(json_value(options, '$.temperature' returning decimal(4,2)), 1);
+  set top_p             = coalesce(json_value(options, '$.top_p' returning decimal(4,2)), 1);
   set n                 = coalesce(json_value(options, '$.n'), 1);
   set stop_token        = json_value(options, '$.stop');
-  set presence_penalty  = coalesce(json_value(options, '$.presence_penalty'), 0);
-  set frequency_penalty = coalesce(json_value(options, '$.frequency_penalty'), 0);
+  set presence_penalty  = coalesce(json_value(options, '$.presence_penalty' returning decimal(4,2)), 0);
+  set frequency_penalty = coalesce(json_value(options, '$.frequency_penalty' returning decimal(4,2)), 0);
   set user_id           = json_value(options, '$.user');
   set seed              = json_value(options, '$.seed');
 
@@ -527,11 +527,11 @@
   end if;
 
   if (json_value(options, '$.temperature') is not null) then
-    set req_body = JSON_UPDATE(req_body, 'SET', '$.temperature', temperature);
+    set req_body = JSON_UPDATE(req_body, 'SET', '$.temperature', cast(temperature as decfloat(16)));
   end if;
 
   if (json_value(options, '$.top_p') is not null) then
-    set req_body = JSON_UPDATE(req_body, 'SET', '$.top_p', top_p);
+    set req_body = JSON_UPDATE(req_body, 'SET', '$.top_p', cast(top_p as decfloat(16)));
   end if;
 
   if (json_value(options, '$.n') is not null) then
@@ -539,11 +539,11 @@
   end if;
 
   if (json_value(options, '$.presence_penalty') is not null) then
-    set req_body = JSON_UPDATE(req_body, 'SET', '$.presence_penalty', presence_penalty);
+    set req_body = JSON_UPDATE(req_body, 'SET', '$.presence_penalty', cast(presence_penalty as decfloat(16)));
   end if;
 
   if (json_value(options, '$.frequency_penalty') is not null) then
-    set req_body = JSON_UPDATE(req_body, 'SET', '$.frequency_penalty', frequency_penalty);
+    set req_body = JSON_UPDATE(req_body, 'SET', '$.frequency_penalty', cast(frequency_penalty as decfloat(16)));
   end if;
 
   if (stop_token is not null) then