Server failing to create PoolableConnectionFactory. Failing with SCRAM-based authentication error.

Start by tracing the PostgreSQL connection setup around Apache Druid's SQLMetadataConnector and BasicDataSourceExt entry points shown in the stack trace, then inspect how the coordinator receives its database password in the Kubernetes deployment. Reproduce the failure with SCRAM-based PostgreSQL authentication and verify that the coordinator starts successfully with the configured password.

Affected Stackable version

nightly

Affected Apache Druid version

28.0.1

Current and expected behavior

A couple notes about our setup:

• We are using postGRES as our metadata storage.
• The metadata storage is using sha-scram-256 for password encryption (this was verified).

When trying the nightly build as of 2024/7/30 12:23PM MDT, all services successfully go online except for the coordinator, which has this error:

org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: Cannot create PoolableConnectionFactory (The server requested SCRAM-based authentication, but no password was provided.)
        at org.skife.jdbi.v2.DBI.open(DBI.java:230) ~[jdbi-2.63.1.jar:2.63.1]
        at org.skife.jdbi.v2.DBI.withHandle(DBI.java:279) ~[jdbi-2.63.1.jar:2.63.1]
        at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:151) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:151) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:161) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.metadata.SQLMetadataConnector.createTable(SQLMetadataConnector.java:212) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.metadata.SQLMetadataConnector.createConfigTable(SQLMetadataConnector.java:388) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.metadata.SQLMetadataConnector.createConfigTable(SQLMetadataConnector.java:715) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.guice.JacksonConfigManagerModule$1.start(JacksonConfigManagerModule.java:58) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:152) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:136) ~[druid-services-28.0.1.jar:28.0.1]
        at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:94) ~[druid-services-28.0.1.jar:28.0.1]
        at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:63) ~[druid-services-28.0.1.jar:28.0.1]
        at org.apache.druid.cli.Main.main(Main.java:112) ~[druid-services-28.0.1.jar:28.0.1]
Caused by: java.sql.SQLException: Cannot create PoolableConnectionFactory (The server requested SCRAM-based authentication, but no password was provided.)
        at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2152) ~[commons-dbcp2-2.0.1.jar:2.0.1]
        at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:1903) ~[commons-dbcp2-2.0.1.jar:2.0.1]
        at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413) ~[commons-dbcp2-2.0.1.jar:2.0.1]
        at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36) ~[jdbi-2.63.1.jar:2.63.1]
        at org.skife.jdbi.v2.DBI.open(DBI.java:212) ~[jdbi-2.63.1.jar:2.63.1]
        ... 18 more
Caused by: org.postgresql.util.PSQLException: The server requested SCRAM-based authentication, but no password was provided.
        at org.postgresql.core.v3.ConnectionFactoryImpl.lambda$doAuthentication$4(ConnectionFactoryImpl.java:838) ~[?:?]
        at org.postgresql.core.v3.AuthenticationPluginManager.withPassword(AuthenticationPluginManager.java:81) ~[?:?]
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:835) ~[?:?]
        at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:203) ~[?:?]
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258) ~[?:?]
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) ~[?:?]
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:263) ~[?:?]
        at org.postgresql.Driver.makeConnection(Driver.java:443) ~[?:?]
        at org.postgresql.Driver.connect(Driver.java:297) ~[?:?]
        at org.apache.druid.metadata.BasicDataSourceExt.lambda$createConnectionFactory$0(BasicDataSourceExt.java:185) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:205) ~[commons-dbcp2-2.0.1.jar:2.0.1]
        at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2162) ~[commons-dbcp2-2.0.1.jar:2.0.1]
        at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2148) ~[commons-dbcp2-2.0.1.jar:2.0.1]
        at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:1903) ~[commons-dbcp2-2.0.1.jar:2.0.1]
        at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413) ~[commons-dbcp2-2.0.1.jar:2.0.1]
        at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36) ~[jdbi-2.63.1.jar:2.63.1]
        at org.skife.jdbi.v2.DBI.open(DBI.java:212) ~[jdbi-2.63.1.jar:2.63.1]
        ... 18 more

Possible solution

No response

Additional context

No response

Environment

AKS

Would you like to work on fixing this bug?

None