UDF: consider returning an enum for `return_type`

Start by reading the UDF trait methods in datafusion/expr/src/udf.rs, especially return_type() and return_field_from_args(), then compare the implementations in datafusion/functions/src/datetime/now.rs. Evaluate whether a unified UdfReturn API and enhanced ReturnFieldArgs can avoid dummy methods without an unacceptable migration burden; done means a reviewed design direction with its compatibility implications documented.

Is your feature request related to a problem or challenge?

UDFs need to implement return_type():

https://github.com/apache/datafusion/blob/a522cd5bcedabd6238a0462b881a7ef4cb5c6f66/datafusion/expr/src/udf.rs#L586

But they can also implement return_field_from_args():

https://github.com/apache/datafusion/blob/a522cd5bcedabd6238a0462b881a7ef4cb5c6f66/datafusion/expr/src/udf.rs#L667

Which supersedes return_type(), but since return_type() still needs to be implemented so usually its made to return an internal error in that case:

https://github.com/apache/datafusion/blob/a522cd5bcedabd6238a0462b881a7ef4cb5c6f66/datafusion/functions/src/datetime/now.rs#L104-L115

This is kind of ugly since:

• Users have to read docs to implement it right (little help from compiler)
• Users might still implement return_type() with an actual implementation even if they implement return_field_from_args(), so return_type() becomes dead code
• Users might accidentally use return_type() in their code for whatever reason, instead of correctly using return_field_from_args()

Describe the solution you'd like

Consider having return type be determined like this:

enum UdfReturn {
    ReturnType(DataType),
    ReturnField(FieldRef),
}

fn returns(&self, args: ReturnFieldArgs) -> Result<UdfReturn> {
    ...
}

• And would need to enhance ReturnFieldArgs to provide just input data types, for ergonomics

This allows us to unify the methods, where simpler UDFs can return the ReturnType variant if they dont care about nullability, field name, metadata, etc.

Describe alternatives you've considered

Don't do this. This is likely to be too breaking of a change, and its hard to migrate towards this path since Rust doesnt allow deprecating trait method implementations (as in it wont show a lint warning). But this was just an idea I had floating in my head and thought it worth recording, and seeing what others thoughts are.

Additional context

In general, I'm just not fond of seeing the approach of having to leave a dummy implementation for trait methods that wont be called. Another example is for UDFs that are meant to always be simplified, so they dont provide an invoke implementation:

https://github.com/apache/datafusion/blob/a522cd5bcedabd6238a0462b881a7ef4cb5c6f66/datafusion/functions/src/datetime/now.rs#L117-L119

It would be nice if we were able to structure the code in a way to avoid this code pattern, but perhaps the ship has sailed.