Stopping the server after a failed startup hangs on Derby

Start with Mirth.startup(), Mirth.shutdown(), and the shutdown-hook registration in Mirth.run(), then inspect the engine and database shutdown calls involved in the Derby path. Reproduce with a fresh Derby install, an invalid SCHEMA_INFO.VERSION, failed startup, and SIGTERM as described. Done means startup failure leaves no engine/database shutdown work that requires a connection, and the process exits promptly without writing the shutdown event; verify both Derby and PostgreSQL behavior where possible.

Summary

After a startup failure such as a MigrationException, the process keeps running (#498). Stopping it normally runs the JVM shutdown hook, which calls Mirth.shutdown() as if the server had started. That stops the engine, which reads from the database, and dispatches a "Server shutdown" event, which a separate listener thread writes to the EVENT table.

• Derby: the hook hangs. Startup already shut Derby down, so the engine waits for a pooled connection that never comes. The process has to be killed with SIGKILL.
• PostgreSQL: the process exits about 2 seconds after SIGTERM. With log_statement = 'all', the only statements from the hook were a read-only SELECT VALUE from CONFIGURATION, run while stopping the engine. No "Server shutdown" event was written in two runs. The event is inserted by the AuditableEventListener thread, so the JVM halting first is timing, not design; on a slower shutdown the insert could reach a database that startup refused to migrate.

MySQL, Oracle and SQL Server were not tested; they share the PostgreSQL path, because stopDatabase() only acts on Derby.

Steps to reproduce

With the default Derby database, from the server's install directory:

1. Start a fresh install once so the database is created, then stop it (Ctrl+C once startup has finished):

   ./oieserver
   

2. With the server stopped, force a migration failure by setting a schema version this release doesn't know:

   java -cp 'server-lib/database/*' org.apache.derby.tools.ij <<'EOF'
   connect 'jdbc:derby:appdata/mirthdb';
   UPDATE SCHEMA_INFO SET VERSION = '99.0.0';
   exit;
   EOF
   

3. Start the server again. Startup fails and the process keeps running (#498):

   ./oieserver
   

   The migration fails with or without the fix for #496 (unrecognized schema version). Only the logged error differs.
4. Press Ctrl+C, or send SIGTERM to the Java process.

Expected

The process exits promptly without touching the database.

Actual (Derby)

The log shows shutting down mirth due to normal request and nothing more. The process was still running more than three minutes later, and SIGKILL was needed. The shutdown hook thread was parked in HikariCP (2.5.1) waiting for a connection:

HikariPool.getConnection
...
DefaultConfigurationController.getProperty(DefaultConfigurationController.java:1029)
DefaultConfigurationController.getChannelDependencies(DefaultConfigurationController.java:1132)
ChannelDependencyServerUtil.getOrderedChannels(ChannelDependencyServerUtil.java:30)
DonkeyEngineController.undeployChannels(DonkeyEngineController.java:462)
DonkeyEngineController.stopEngine(DonkeyEngineController.java:275)
Mirth.stopEngine(Mirth.java:461)
Mirth.shutdown(Mirth.java:417)
Mirth$ShutdownHook.run(Mirth.java:531)

No new row was written to EVENT on Derby, and SCHEMA_INFO was unchanged.

Cause

The MigrationException handler in Mirth.startup() calls stopDatabase() (which only shuts down Derby) and returns. The ShutdownHook registered in Mirth.run() then calls shutdown() unconditionally. shutdown() assumes a started server: it stops the engine, opens a database session to dispatch the "Server shutdown" event, and stops the web server, plugins and database.

On Derby the wait never ends because HikariConnectionPool sets connectionTimeout to 0, which HikariCP treats as no timeout. Once Derby is shut down, no connection can be created, so the hook waits forever.

A port conflict doesn't reach this path, because Mirth.run() returns before the hook is registered.

Suggested fix

Track whether startup completed, and have shutdown() skip stopping the engine and dispatching the event when it did not. Fix this together with #498. Making the process exit on a startup failure means calling System.exit, which runs this hook, so on Derby that fix would hang here unless this one lands with it.

🤖 Drafted with Claude Code