[BUG] Docker: OBP-API fails to start with 503 error, caused by ExceptionInInitializerError in ErrorMessages

Bắt đầu với ErrorMessages.scala ở dòng 119 và lần theo quá trình khởi tạo được truy cập từ code.api.Constant$ HostName1 trong constant.scala:25. Tái hiện lỗi bằng docker-compose.yml của OBP-Docker và default.props được mount, sau đó sử dụng log của container để xác định vì sao quá trình khởi động dẫn đến ExceptionInInitializerError; hoàn thành khi container API khởi động mà không có 503 và báo cáo rõ ràng các lỗi cấu hình.

Issue Description

Environment:

• OS: macOS (Darwin)
• Docker: Docker Desktop
• Project: OBP-Docker
• API Image: openbankproject/obp-api:latest (and other stable versions
  attempted)
• Database Image: postgres:14

Summary:

When following the standard OBP-Docker setup, the obp-api container
consistently fails to initialize its web application, resulting in an HTTP
503 Service Unavailable error.

The logs repeatedly show a misleading error message: ERROR
net.liftweb.util.Props - Failed to find a properties file. However, extensive
debugging has shown that the root cause is a
java.lang.ExceptionInInitializerError which originates from a
NullPointerException within the code.api.util.ErrorMessages class during its
static initialization.

Key Stack Trace:

1 java.lang.ExceptionInInitializerError
2 at code.api.Constant$.$anonfun$HostName1(constant.scala:25)
3 ...
4 Caused by: java.lang.NullPointerException
5 at code.api.util.ErrorMessages$.(ErrorMessages.scala:
119)
6 at code.api.util.ErrorMessages$.(ErrorMessages.scala)
7 ...

Troubleshooting Steps Performed:

We have systematically diagnosed and resolved a series of configuration
issues, but the application still fails to start. This suggests the issue is
internal to the application itself.

Here is a summary of our debugging journey:

1. Initial State: Standard docker-compose up resulted in a 503 error. Logs
   showed Failed to find a properties file.
2. Fix #1 (File Mount): We corrected the docker-compose.yml to properly mount
   the local ./api/default.props file to /props/default.props inside the
   container.
3. New Error: After Fix #1, the error changed to
   org.postgresql.util.PSQLException: FATAL: database "obp" does not exist.
   This confirmed the props file was now being read correctly.
4. Fix #2 (Database Reset): We resolved the database creation issue by stopping
   the containers, deleting the ./pgdata persistent volume, and restarting
   docker-compose. This allowed the PostgreSQL container to initialize correctly
   and create the obp database.
5. Error Regression: After Fix #2, the error unexpectedly reverted back to the
   original Failed to find a properties file and the
   ExceptionInInitializerError, even though the database was now available.
6. Verification: We used docker exec obp-api ls -l /props/default.props to
   confirm the file exists inside the container and has correct jetty:jetty
   ownership and world-readable permissions. This proves the "file not found"
   error is misleading.
7. Fix #3 (Isolate Config): To rule out conflicts between file-based and
   environment-variable-based configuration, we commented out all db.driver and
   db.url properties from default.props, forcing the application to use the
   DB_HOST, DB_USER, etc., environment variables. The exact same
   ExceptionInInitializerError persisted.
8. Final Attempt (Change Version): Changed the image tag from latest to another
   version, but the startup failed with the same error pattern.

Conclusion:

After exhausting all external configuration possibilities, the evidence
strongly suggests this is not a user setup error. It is likely a bug in the
application's startup sequence or an undocumented, critical property that is
required for the ErrorMessages class to initialize. The application is not
robust enough to fail with a clear error message about what property is
missing.

Full obp-api container logs
obp-api | /opt/java/openjdk/bin/java -Djava.io.tmpdir=/tmp
/jetty -Djetty.home=/usr/local/jetty -Djetty.base=/var/lib/
jetty -Djava.io.tmpdir=/tmp/jetty -cp /usr/local/jetty/lib/
mail/javax.mail.glassfish-1.4.1.v201005082020.jar:/var/lib/
jetty/resources:/usr/local/jetty/lib/servlet-api-3.1.jar:
/usr/local/jetty/lib/jetty-schemas-3.1.jar:/usr/local/jetty
/lib/jetty-http-9.4.58.v20250814.jar:/usr/local/jetty/lib
/jetty-server-9.4.58.v20250814.jar:/usr/local/jetty/lib/
jetty-xml-9.4.58.v20250814.jar:/usr/local/jetty/lib
/jetty-util-9.4.58.v20250814.jar:/usr/local/jetty/lib/jetty
-io-9.4.58.v20250814.jar:/usr/local/jetty/lib
/jetty-jndi-9.4.58.v20250814.jar:/usr/local/jetty/lib/jetty
-security-9.4.58.v20250814.jar:/usr/local/jetty/lib
/transactions/javax.transaction-api-1.3.jar:/usr/local
/jetty/lib/jetty-servlet-9.4.58.v20250814.jar:/usr/local/
jetty/lib/jetty-webapp-9.4.58.v20250814.jar:/usr/local
/jetty/lib/jetty-plus-9.4.58.v20250814.jar:/usr/local/jetty
/lib/jetty-annotations-9.4.58.v20250814.jar:/usr/local
/jetty/lib/annotations/asm-9.7.1.jar:/usr/local/jetty/lib
/annotations/asm-analysis-9.7.1.jar:/usr/local/jetty/lib
/annotations/asm-commons-9.7.1.jar:/usr/local/jetty/lib
/annotations/asm-tree-9.7.1.jar:/usr/local/jetty/lib
/annotations/javax.annotation-api-1.3.2.jar:/usr/local
/jetty/lib/apache-jsp/org.eclipse.jdt.ecj-3.26.0.jar:/usr/
local/jetty/lib/apache-jsp/org.eclipse.jetty.apache-jsp-9.4
.58.v20250814.jar:/usr/local/jetty/lib/apache-jsp/
org.mortbay.jasper.apache-el-8.5.100.jar:/usr/local/jetty/
lib/apache-jsp/org.mortbay.jasper.apache-jsp-8.5.100.jar:
/usr/local/jetty/lib/apache-jstl/org.apache.taglibs.taglibs
-standard-impl-1.2.5.jar:/usr/local/jetty/lib/apache-jstl/
org.apache.taglibs.taglibs-standard-spec-1.2.5.jar:/usr/
local/jetty/lib/jetty-client-9.4.58.v20250814.jar:/usr
/local/jetty/lib/jetty-deploy-9.4.58.v20250814.jar:/usr/
local/jetty/lib/websocket/javax.websocket-api-1.0.jar:/usr/
local/jetty/lib/websocket/javax-websocket-client-impl-9.4.
58.v20250814.jar:/usr/local/jetty/lib/websocket/javax-
websocket-server-impl-9.4.58.v20250814.jar:/usr/local
/jetty/lib/websocket/websocket-api-9.4.58.v20250814.jar:
/usr/local/jetty/lib/websocket/websocket-client-9.4.58
.v20250814.jar:/usr/local/jetty/lib/websocket/websocket-
common-9.4.58.v20250814.jar:/usr/local/jetty/lib/websocket/
websocket-server-9.4.58.v20250814.jar:/usr/local/jetty/lib
/websocket/websocket-servlet-9.4.58.v20250814.jar
org.eclipse.jetty.xml.XmlConfiguration at.least.java9=true
java.io.tmpdir=/tmp/jetty java.version=11.0.28
java.version.major=11 java.version.micro=28
java.version.minor=0 java.version.platform=11 jetty.base=/
var/lib/jetty jetty.base.uri=file:///var/lib/jetty
jetty.home=/usr/local/jetty
jetty.home.uri=file:///usr/local/jetty
runtime.feature.alpn=true
/usr/local/jetty/etc/jetty-bytebufferpool.xml
/usr/local/jetty/etc/jetty-threadpool.xml
/usr/local/jetty/etc/jetty.xml
/usr/local/jetty/etc/jetty-webapp.xml
/usr/local/jetty/etc/jetty-plus.xml
/usr/local/jetty/etc/jetty-annotations.xml
/usr/local/jetty/etc/jetty-deploy.xml
/usr/local/jetty/etc/jetty-http.xml
2 obp-api | 2025-10-29 02:41:17.685:INFO::main: Logging
initialized @424ms to org.eclipse.jetty.util.log.StdErrLog
3 obp-api | 2025-10-29 02:41:18.169:INFO:oejs.Server:main:
jetty-9.4.58.v20250814; built: 2025-08-14T02:28:49.637Z;
git: 8f1440587e9e4ae7db3d74cf205643f3d707148d; jvm 11.0.28
+6
4 obp-api | 2025-10-29 02:41:18.213
:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [
file:///var/lib/jetty/webapps/] at interval 1
5 obp-api | 2025-10-29
02:41:23.815:WARN:oeja.ClassInheritanceHandler:qtp180065951
9-18:
6 obp-api | java.lang.NullPointerException
7 obp-api | at javabase
/java.util.concurrent.ConcurrentHashMap.get
(ConcurrentHashMap.java:936)
8 obp-api | at org.eclipse.jetty.annotations
.ClassInheritanceHandler.addToInheritanceMap
(ClassInheritanceHandler.java:75)
9 obp-api | at org.eclipse.jetty.annotations
.ClassInheritanceHandler.handle(ClassInheritanceHandler.
java:62)
10 obp-api | at
org.eclipse.jetty.annotations.AnnotationParser
$MyClassVisitor.visit(AnnotationParser.java:488)
11 obp-api | at org.objectwebasm.ClassReader.accept
(ClassReader.java:570)
12 obp-api | at org.objectwebasm.ClassReader.accept
(ClassReader.java:425)
13 obp-api | at org.eclipse.jetty.annotations
.AnnotationParser.scanClass(AnnotationParser.java:933)
14 obp-api | at org.eclipse.jetty.annotations
.AnnotationParser.parseJarEntry(AnnotationParser.java:917)
15 obp-api | at org.eclipse.jetty.annotations.
AnnotationParser.lambda$parseJar$2(AnnotationParser.java:
876)
16 obp-api | at java.base/java.util.TreeMa$ValueSpliterator
.forEachRemaining(TreeMap.java:2890)
17 obp-api | at java.base/java.util.stream.ReferencePipeline
$Head.forEach(ReferencePipeline.java:658)
18 obp-api | at org.eclipse.jetty.annotations
.AnnotationParser.parseJar(AnnotationParser.java:872)
19 obp-api | at org.eclipse.jetty.annotations
.AnnotationParser.parse(AnnotationParser.java:835)
20 obp-api | at
org.eclipse.jetty.annotations.AnnotationConfiguration
$ParserTask.call(AnnotationConfiguration.java:159)
21 obp-api | at
org.eclipse.jetty.annotations.AnnotationConfiguration$1
.run(AnnotationConfiguration.java:455)
22 obp-api | at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:883)
23 obp-api | at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run
(QueuedThreadPool.java:1034)
24 obp-api | at
java.base/java.lang.Thread.run(Thread.java:829)
25 obp-api | 2025-10-29
02:41:23.886:WARN:oeja.ClassInheritanceHandler:qtp180065951
9-20:
26 obp-api | java.lang.NullPointerException
27 obp-api | at javabase
/java.util.concurrent.ConcurrentHashMap.get
(ConcurrentHashMap.java:936)
28 obp-api | at org.eclipse.jetty.annotations
.ClassInheritanceHandler.addToInheritanceMap
(ClassInheritanceHandler.java:75)
29 obp-api | at org.eclipse.jetty.annotations
.ClassInheritanceHandler.handle(ClassInheritanceHandler.
java:62)
30 obp-api | at
org.eclipse.jetty.annotations.AnnotationParser
$MyClassVisitor.visit(AnnotationParser.java:488)
31 obp-api | at org.objectwebasm.ClassReader.accept
(ClassReader.java:570)
32 obp-api | at org.objectwebasm.ClassReader.accept
(ClassReader.java:425)
33 obp-api | at org.eclipse.jetty.annotations
.AnnotationParser.scanClass(AnnotationParser.java:933)
34 obp-api | at org.eclipse.jetty.annotations
.AnnotationParser.parseJarEntry(AnnotationParser.java:917)
35 obp-api | at org.eclipse.jetty.annotations.
AnnotationParser.lambda$parseJar$2(AnnotationParser.java:
876)
36 obp-api | at java.base/java.util.TreeMa$ValueSpliterator
.forEachRemaining(TreeMap.java:2890)
37 obp-api | at java.base/java.util.stream.ReferencePipeline
$Head.forEach(ReferencePipeline.java:658)
38 obp-api | at org.eclipse.jetty.annotations
.AnnotationParser.parseJar(AnnotationParser.java:872)
39 obp-api | at org.eclipse.jetty.annotations
.AnnotationParser.parse(AnnotationParser.java:835)
40 obp-api | at
org.eclipse.jetty.annotations.AnnotationConfiguration
$ParserTask.call(AnnotationConfiguration.java:159)
41 obp-api | at
org.eclipse.jetty.annotations.AnnotationConfiguration$1
.run(AnnotationConfiguration.java:455)
42 obp-api | at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:883)
43 obp-api | at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run
(QueuedThreadPool.java:1034)
44 obp-api | at
java.base/java.lang.Thread.run(Thread.java:829)
45 obp-api | 2025-10-29
02:41:26.438:INFO:oeja.AnnotationConfiguration:main:
Scanning elapsed time=5198ms
46 obp-api | 2025-10-29 02:41:26.706:INFO:oejs.session:main:
DefaultSessionIdManager workerName=node0
47 obp-api | 2025-10-29 02:41:26.707:INFO:oejs.session:main:
No SessionScavenger set, using defaults
48 obp-api | 2025-10-29 02:41:26.707:INFO:oejs.session:main:
node0 Scavenging every 600000ms
49 obp-api | 2025-10-29 02:41:27 main Props [ERROR] Failed to
find a properties file (but properties were accessed).
Searched: /props/jetty.5b019d435781.props, /props/jetty.
props, /props/5b019d435781.props, /props/default.props,
/jetty.5b019d435781.props, /jetty.props,
/5b019d435781.props, /default.props
50 obp-api | 2025-10-29 02:41:27 main Constant$ [INFO]
Instantiating Constants
51 obp-api | 2025-10-29 02:41:27.840
:WARN:oejw.WebAppContext:main: Failed startup of context
o.e.j.w.WebAppContext@2e8c1c9b{ROOT,/,file:///
tmp/jetty-0_0_0_0-8080-ROOT_war-_-any-16333074229383268091
/webapp/,UNAVAILABLE}{/var/lib/jetty/webapps/ROOT.war}
52 obp-api | java.lang.ExceptionInInitializerError
53 obp-api | at code.apiConstant$.$anonfun$HostName$1
(constant.scala:25)
54 obp-api | at net.liftweb.common.EmptyBox
.openOrThrowException(Box.scala:910)
55 obp-api | at net.liftweb.common.EmptyBox
.openOrThrowException(Box.scala:905)
56 obp-api | at code.api.Constant$.(constant.scala:25)
57 obp-api | atcode.api.Constant$.(constant.scala)
58 obp-api | at code.util.Helper$.getHostname(Helper.scala:
275)
59 obp-api | at code.util.Helper$MdcLoggable.$init$(Helper.
scala:415)
60 obp-api | at code.api.util.APIUtil$.(APIUtil.scala:
117)
61 obp-api | atcode.api.util.APIUtil$.(APIUtil.scala
)
62 obp-api | at code.util.SecureLogging$.conditionalPattern
(SecureLogging.scala:25)
63 obp-api | at code.util.SecureLogging$.sensitivePatterns$
lzycompute(SecureLogging.scala:35)
64 obp-api | at code.util.SecureLogging$.sensitivePatterns
(SecureLogging.scala:31)
65 obp-api | at code.util.SecureLogging$.maskSensitive
(SecureLogging.scala:103)
66 obp-api | at code.utilHelper$MdcLoggable$$anon$3.info(
Helper.scala:346)
67 obp-api | at bootstrap.liftweb.Boot.initiate(Boot.scala:
227)
68 obp-api | at code.util.Helper$MdcLoggable.$init$(Helper.
scala:414)
69 obp-api | at bootstrap.liftweb.Boot.init>(Boot.scala:165)
70 obp-api | at javabase/jdk.internal
.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
71 obp-api | at javabase/jdk.internal
.reflect.NativeConstructorAccessorImpl.newInstance(NativeCo
nstructorAccessorImpl.java:62)
72 obp-api | at javabase/jdk.internal
.reflect.DelegatingConstructorAccessorImpl.newInstance(Dele
gatingConstructorAccessorImpl.java:45)
73 obp-api | at
java.base/java.lang.reflect.Constructor.newInstance(Constru
ctor.java:490)
74 obp-api | at java.base/java.langClass.newInstance(Class
.java:584)
75 obp-api | at net.liftweb.http.DefaultBootstrap$.boot
(LiftRules.scala:2249)
76 obp-api | at net.liftweb.http.provider.HTTPProvider
.bootLift(HTTPProvider.scala:88)
77 obp-api | at net.liftweb.http.provider.HTTPProvider
.bootLift$(HTTPProvider.scala:83)
78 obp-api | at net.liftweb.http.LiftFilter.bootLift
(LiftServlet.scala:1063)
79 obp-api | at
net.liftweb.http.provider.servlet.ServletFilterProvider.
init(ServletFilterProvider.scala:40)
80 obp-api | at
net.liftweb.http.provider.servlet.ServletFilterProvider.
init$(ServletFilterProvider.scala:35)
81 obp-api | at net.liftweb.http.LiftFilterinit
(LiftServlet.scala:1063)
82 obp-api | at org.eclipse.jetty.servlet.FilterHolder
.initialize(FilterHolder.java:140)
83 obp-api | at org.eclipse.jetty.servletServletHandler
.lambda$initialize$0(ServletHandler.java:750)
84 obp-api | at java.base/java.util.Spliterators
$ArraySpliterator.forEachRemaining(Spliterators.java:948)
85 obp-api | at java.base/java.util.stream.Streams
$ConcatSpliterator.forEachRemaining(Streams.java:734)
86 obp-api | at java.base/java.util.stream.ReferencePipeline
$Head.forEach(ReferencePipeline.java:658)
87 obp-api | at org.eclipse.jetty.servlet.ServletHandler
.initialize(ServletHandler.java:774)
88 obp-api | at org.eclipse.jetty.servlet
.ServletContextHandler.startContext(ServletContextHandler.
java:379)
89 obp-api | at org.eclipse.jetty.webapp.WebAppContext
.startWebapp(WebAppContext.java:1449)
90 obp-api | at org.eclipse.jetty.webapp.WebAppContext
.startContext(WebAppContext.java:1414)
91 obp-api | at org.eclipse.jetty.server.handler
.ContextHandler.doStart(ContextHandler.java:916)
92 obp-api | at org.eclipse.jetty.servlet
.ServletContextHandler.doStart(ServletContextHandler.java:
288)
93 obp-api | at org.eclipse.jetty.webapp.WebAppContext
.doStart(WebAppContext.java:524)
94 obp-api | at org.eclipse.jetty.util.component
.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
95 obp-api | at org.eclipse.jetty.deploy.bindings
.StandardStarter.processBinding(StandardStarter.java:46)
96 obp-api | at org.eclipse.jetty.deploy.AppLifeCycle
.runBindings(AppLifeCycle.java:188)
97 obp-api | at org.eclipse.jetty.deploy.DeploymentManager
.requestAppGoal(DeploymentManager.java:517)
98 obp-api | at org.eclipse.jetty.deploy.DeploymentManager
.addApp(DeploymentManager.java:157)
99 obp-api | at org.eclipse.jetty.deploy.providers
.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173
)
100 obp-api | at org.eclipse.jetty.deploy.providers
.WebAppProvider.fileAdded(WebAppProvider.java:442)
101 obp-api | at
org.eclipse.jetty.deploy.providers.ScanningAppProvider$1
.fileAdded(ScanningAppProvider.java:66)
102 obp-api | at org.eclipse.jetty.util.Scanner.reportAddition
(Scanner.java:785)
103 obp-api | at org.eclipse.jetty.util.Scanner
.reportDifferences(Scanner.java:754)
104 obp-api | at org.eclipse.jetty.util.Scanner.scan(Scanner.
java:641)
105 obp-api | at org.eclipse.jetty.util.Scanner.doStart
(Scanner.java:540)
106 obp-api | at org.eclipse.jetty.util.component
.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
107 obp-api | at org.eclipse.jetty.deploy.providers
.ScanningAppProvider.doStart(ScanningAppProvider.java:146)
108 obp-api | at org.eclipse.jetty.util.component
.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
109 obp-api | at org.eclipse.jetty.deploy.DeploymentManager
.startAppProvider(DeploymentManager.java:605)
110 obp-api | at org.eclipse.jetty.deploy.DeploymentManager
.doStart(DeploymentManager.java:252)
111 obp-api | at org.eclipse.jetty.util.component
.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
112 obp-api | at org.eclipse.jetty.util.component
.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
113 obp-api | at org.eclipse.jetty.serverServer.start(Server
.java:423)
114 obp-api | at org.eclipse.jetty.util.component
.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
115 obp-api | at org.eclipse.jetty.server.handler
.AbstractHandler.doStart(AbstractHandler.java:97)
116 obp-api | at org.eclipse.jetty.serverServer.doStart(
Server.java:387)
117 obp-api | at org.eclipse.jetty.util.component
.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
118 obp-api | at org.eclipse.jetty.xml.XmlConfiguration.lambda
$main$3(XmlConfiguration.java:1931)
119 obp-api | at
java.base/java.security.AccessController.doPrivileged(Nativ
e Method)
120 obp-api | at org.eclipse.jetty.xml.XmlConfiguration.main
(XmlConfiguration.java:1881)
121 obp-api | Caused by: java.lang.NullPointerException
122 obp-api | at code.api.util.ErrorMessages$.(ErrorMessages.scala:119)
123 obp-api | atcode.api.util.ErrorMessages
$.(ErrorMessages.scala)
124 obp-api | ... 68 more
125 obp-api | 2025-10-29 02:41:27.862
:INFO:oejs.AbstractConnector:main: Started
ServerConnector@f25e919{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
126 obp-api | 2025-10-29 02:41:27.871:INFO:oejs.Server:main:
Started @10613ms

docker-compose.yml