Caught AR validation errors are resulting in `error` spans

Our app uses ActiveRecord validations (example), and uses some create! calls in the code (example controller action, and create!).

Then a record fails the validation, an exception is thrown, and is then caught and returned as an HTTP error, with the failed validations in the body (here).

This causes the OTEL instrumentation to generate traces where the User.create! span has an error attached, but the root span does not:

This causes tools like Sentry or Datadog to create a new entry in their error tracking system, even if everything is handled in the app and the error is not actionable.

Using exceptions for such control flow is a common pattern in Rails, and as an application developer or operator, I expect that errors are things that needs to be reviewed and fixed, which is not the case here.

Note: there is a relevant discussion in the Slack channel

Possible discussed solutions are marking some exceptions (such as ActiveRecord::RecordInvalid or ActiveRecord:: RecordNotFound) as non-exceptional in the AR Validations instrumentation, or having a setting to prevent those spans to be errors entirely, allowing the app developer to explicitly indicate that those exceptions are handled correctly by the app (and if thats not the case, the root span should be an error anyway).