[Bug] Division by zero in gptq_quantize_matrix

Description

gptq_quantize_matrix (keras/src/quantizers/gptq.py), Keras computes the error term by dividing by the diagonal element of the block's inverse Hessian:

current_block_influence = block_inv_hessian[block_idx, block_idx]
err = ops.divide(
    ops.subtract(weight_column, dequantized_col),
    current_block_influence,
)

However, there is no guard such as a small epsilon on current_block_influence. If the diagonal of the inverse Hessian underflows to 0.0 (or becomes very small) under some backends / precision settings, this division will result in inf or NaN propagating to the other weight updates and corrupting the entire weight matrix.

Reproduction Script

import numpy as np
import keras
from keras.src.quantizers.gptq import gptq_quantize_matrix
from keras.src.quantizers.quantizers import GPTQQuantizer
from keras.src.quantizers.gptq_config import GPTQConfig

out_features, in_features = 4, 4
weights = keras.ops.ones((out_features, in_features), dtype="float32")

# Construct an inverse Hessian where one diagonal element is zero
inv_hessian = np.eye(in_features, dtype=np.float32)
inv_hessian[2, 2] = 0.0  # Underflow / division-by-zero trigger
inv_hessian = keras.ops.convert_to_tensor(inv_hessian)

# Initialize GPTQ config and quantizer to provide bound find_params callable
config = GPTQConfig(dataset=None, tokenizer=None, weight_bits=4, group_size=-1)
quantizer = GPTQQuantizer(config)

# Run quantization
quantized, scale, zero, g_idx = gptq_quantize_matrix(
    weights,
    inv_hessian,
    blocksize=2,
    group_size=-1,
    compute_scale_zero=quantizer.find_params
)

print("Quantized weights:\n", quantized)

Actual Output

Depending on your backend version, and hardware ( CPU/GPU ) casting NaN to int will output either 0 or underflow bounds ( like -2147483648 ) for the columns after the zero influence feature.

Quantized weights:
 tf.Tensor(
[[7 7 7 0]
 [7 7 7 0]
 [7 7 7 0]
 [7 7 7 0]], shape=(4, 4), dtype=int32)

The last column is quantized incorrectly to 0 or -2147483648 instead of the original weight scale because the zero-influence diagonal element propagates NaN values to all subsequent columns during the error feedback loop).

Expected Behavior

A small epsilon should be used to guard the division to avoid propagation of NaN/overflow. This is to make sure that quantization process completes without numeric failures even if the diagonal elements underflow or become zero.

Impact

• Cascading NaNs/Infs: Silent weights corruption during quantization when calibration data leads to a singular/semi-singular Hessian.
• Quantization failure: Quantized weights are populated with corrupted/underflow integer values, rendering the model unusable.

Environment

Keras: 3.15.1
Python: 3.12.13
Backend: Multi-backend 
Google Cloab: Linux-6.6.122+-x86_64-with-glibc2.35

Note

While Hessian dampening stabilizes the Hessian matrix before inversion, the diagonal elements of the inverse Hessian matrix can still underflow or get close to zero. Protecting the division step in the inner loop is essential for numerical robustness.