[Issues] When Initializing a new Cloudberry cluster we are encountering issues

The log points at the dtx recovery background worker (signal 11 in postmaster.c) and the ic-proxy process, with the crash address in postgres. The crashes follow gp_add_segment_mirror calls during gpinitsystem, so start with the segment-mirror registration path and the core dumps from the failing run. Done would mean a reproduction or a root-cause fix for the segfault, which the report does not yet provide.

Apache Cloudberry version

Apache Cloudberry 2.1.0-incubating build bdf90c5518f

What happened

We were attempting to initialize a new Cloudberry cluster and the gpinitsystem failed,

In the gpdb logs we are observing the following
background worker ""dtx recovery process"" (PID 2371538) was terminated by signal 11
gpAdmin logs
psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: the database system is in recovery mode
DETAIL: last replayed record at 0/0
var log messages
general protection fault ip:e1d282 sp:7ffed4b01c60 error:0 in postgres[6e6000+a69000]
postgres[2374077]: segfault at 4 ip 0000000000e1d282 sp 00007ffed4b01e10 error 4 in postgres[6e6000+a69000] likely on CPU 28 (core 4, socket 0)

And continous coredump files being created until we terminate postgres proces

What you think should happen instead

Should be a clean install. gpinitsystem was successful 3 other CB clusters (2 physical hardware and 1 VM)

How to reproduce

We can only reproduce on this one set of hardware, Any help would be appreciated on what is causing the failure

Operating System

NAME="Red Hat Enterprise Linux" VERSION="9.7 (Plow)"

Anything else

gpinitsystem_20261002.log

20261002:22:27:16:2201694 gpinitsystem:pe1uaer5gpm01:gpadmin-[INFO]:-Start Function ERROR_CHK
20261002:22:27:16:2201694 gpinitsystem:pe1uaer5gpm01:gpadmin-[INFO]:-Successfully completed failed to register mirror for contentid=91
20261002:22:27:16:2201694 gpinitsystem:pe1uaer5gpm01:gpadmin-[INFO]:-End Function ERROR_CHK
20261002:22:27:16:2201694 gpinitsystem:pe1uaer5gpm01:gpadmin-[INFO]:-Start Function ERROR_CHK
20261002:22:27:16:2201694 gpinitsystem:pe1uaer5gpm01:gpadmin-[INFO]:-Successfully completed failed to register mirror for contentid=92
20261002:22:27:16:2201694 gpinitsystem:pe1uaer5gpm01:gpadmin-[INFO]:-End Function ERROR_CHK
psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: the database system is in recovery mode
DETAIL: last replayed record at 0/0
20261002:22:27:17:2201694 gpinitsystem:pe1uaer5gpm01:gpadmin-[INFO]:-Start Function ERROR_CHK
20261002:22:27:17:2201694 gpinitsystem:pe1uaer5gpm01:gpadmin-[INFO]:-End Function ERROR_CHK
20261002:22:27:17:2201694 gpinitsystem:pe1uaer5gpm01:gpadmin-[INFO]:-Start Function ERROR_EXIT
20261002:22:27:17:2201694 gpinitsystem:pe1uaer5gpm01:gpadmin-[FATAL]:-Failed to complete failed to register mirror for contentid=93 Script Exiting!

coordinator log

2026-10-02 22:27:16.737883 PDT,,,p2371526,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","background worker ""dtx recovery process"" (PID 2371538) was terminated by signal 11: Segmentation fault",,,,,,,0,,"postmaster.c",4285,
2026-10-02 22:27:16.738047 PDT,,,p2371526,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","terminating any other active server processes",,,,,,,0,,"postmaster.c",4000,
2026-10-02 22:27:16.738063 PDT,,,p2371541,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","ic-proxy: received signal 3",,,,,,,0,,"ic_proxy_main.c",484,
2026-10-02 22:27:16.738071 PDT,,,p2371541,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","ic-proxy: server closing",,,,,,,0,,"ic_proxy_main.c",585,
2026-10-02 22:27:16.738078 PDT,,,p2371541,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","ic-proxy: server closed with code 1",,,,,,,0,,"ic_proxy_main.c",599,
2026-10-02 22:27:16.739198 PDT,,,p2371526,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","background worker ""ic proxy process"" (PID 2371541) exited with exit code 1",,,,,,,0,,"postmaster.c",4264,
2026-10-02 22:27:16.739446 PDT,,,p2371526,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","all server processes terminated; reinitializing",,,,,,,0,,"postmaster.c",4576,
2026-10-02 22:27:16.775137 PDT,,,p2374065,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","database system was interrupted; last known up at 2026-10-02 22:27:11 PDT",,,,,,,0,,"xlog.c",6887,
2026-10-02 22:27:16.775166 PDT,,,p2374065,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","Synchronization of the wal directory starts.",,,,,,,0,,"fd.c",3453,
2026-10-02 22:27:16.775207 PDT,,,p2374065,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","synchronization of the wal directory finishes.",,,,,,,0,,"fd.c",3455,
2026-10-02 22:27:16.775475 PDT,,,p2374065,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","database system was not properly shut down; automatic recovery in progress",,,,,,,0,,"xlog.c",7456,
2026-10-02 22:27:16.775842 PDT,,,p2374065,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","redo starts at 0/4972CD0",,,,,,,0,,"xlog.c",7736,
2026-10-02 22:27:16.776497 PDT,,,p2374065,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","invalid record length at 0/497D068: wanted 24, got 0",,,,,,,0,,"xlog.c",4496,
2026-10-02 22:27:16.776504 PDT,,,p2374065,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","redo done at 0/497D038 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s",,,,,,,0,,"xlog.c",8033,
2026-10-02 22:27:16.776762 PDT,,,p2374065,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","end of transaction log location is 0/497D068",,,,,,,0,,"xlog.c",8122,
2026-10-02 22:27:16.777350 PDT,,,p2374065,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","checkpoint starting: end-of-recovery immediate",,,,,,,0,,"xlog.c",9315,
2026-10-02 22:27:16.778146 PDT,,,p2374065,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","checkpoint complete: wrote 10 buffers (0.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.002 s; sync files=9, longest=0.001 s, average=0.001 s; distance=40 kB, estimate=40 kB",,,,,,,0,,"xlog.c",9396,
2026-10-02 22:27:16.778157 PDT,,,p2374065,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","latest completed transaction id is 1138 and next transaction id is 1139",,,,,,,0,,"xlog.c",8432,
2026-10-02 22:27:16.778221 PDT,,,p2374065,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","database system is ready",,,,,,,0,,"xlog.c",8564,
2026-10-02 22:27:16.781696 PDT,,,p2371526,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","PostgreSQL 14.4 (Apache Cloudberry 2.1.0-incubating build bdf90c5518f) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11), 64-bit compiled on Apr 26 2026 20:17:15",,,,,,,0,,"postmaster.c",3569,
2026-10-02 22:27:16.781707 PDT,,,p2371526,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","database system is ready to accept connections","PostgreSQL 14.4 (Apache Cloudberry 2.1.0-incubating build bdf90c5518f) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11), 64-bit compiled on Apr 26 2026 20:17:15",,,,,,0,,"postmaster.c",3571,
2026-10-02 22:27:16.787863 PDT,,,p2374078,th-270645056,,,,0,con4,,seg-1,,,,sx1,"LOG","00000","initialized 1 resource queues",,,,,,,0,,"resscheduler.c",267,
2026-10-02 22:27:16.797428 PDT,"gpadmin","template1",p2374081,th-270645056,"[local]",,2026-10-02 22:27:16 PDT,0,con5,cmd1,seg-1,,,,sx1,"LOG","00000","statement: select pg_catalog.gp_add_segment_mirror(90::int2, 'pe1uaer5gps33.res.prod.global', 'pe1uaer5gps33.res.prod.global', 7002, '/data1/cb/mirror/gpseg90');",,,,,,,0,,"postgres.c",1715,
2026-10-02 22:27:16.855311 PDT,"gpadmin","template1",p2374119,th-270645056,"[local]",,2026-10-02 22:27:16 PDT,0,con6,cmd1,seg-1,,,,sx1,"LOG","00000","statement: select pg_catalog.gp_add_segment_mirror(91::int2, 'pe1uaer5gps34.res.prod.global', 'pe1uaer5gps34.res.prod.global', 7003, '/data1/cb/mirror/gpseg91');",,,,,,,0,,"postgres.c",1715,
2026-10-02 22:27:16.910881 PDT,"gpadmin","template1",p2374155,th-270645056,"[local]",,2026-10-02 22:27:16 PDT,0,con7,cmd1,seg-1,,,,sx1,"LOG","00000","statement: select pg_catalog.gp_add_segment_mirror(92::int2, 'pe1uaer5gps35.res.prod.global', 'pe1uaer5gps35.res.prod.global', 7004, '/data2/cb/mirror/gpseg92');",,,,,,,0,,"postgres.c",1715,
2026-10-02 22:27:16.960131 PDT,,,p2371526,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","background worker ""dtx recovery process"" (PID 2374077) was terminated by signal 11: Segmentation fault",,,,,,,0,,"postmaster.c",4285,
2026-10-02 22:27:16.960156 PDT,,,p2371526,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","terminating any other active server processes",,,,,,,0,,"postmaster.c",4000,
2026-10-02 22:27:16.961508 PDT,,,p2371526,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","all server processes terminated; reinitializing",,,,,,,0,,"postmaster.c",4576,
2026-10-02 22:27:16.996475 PDT,,,p2374196,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","database system was interrupted; last known up at 2026-10-02 22:27:16 PDT",,,,,,,0,,"xlog.c",6887,
2026-10-02 22:27:16.996503 PDT,,,p2374196,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","Synchronization of the wal directory starts.",,,,,,,0,,"fd.c",3453,
2026-10-02 22:27:16.996543 PDT,,,p2374196,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","synchronization of the wal directory finishes.",,,,,,,0,,"fd.c",3455,
2026-10-02 22:27:16.996658 PDT,"gpadmin","template1",p2374197,th-270645056,"[local]",,2026-10-02 22:27:16 PDT,0,,,seg-1,,,,,"FATAL","57P03","the database system is in recovery mode","last replayed record at 0/0",,,,,,0,,"postmaster.c",2786,
2026-10-02 22:27:16.996779 PDT,,,p2374196,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","database system was not properly shut down; automatic recovery in progress",,,,,,,0,,"xlog.c",7456,
2026-10-02 22:27:16.997141 PDT,,,p2374196,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","redo starts at 0/497D0F0",,,,,,,0,,"xlog.c",7736,
2026-10-02 22:27:16.997715 PDT,,,p2374196,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","invalid record length at 0/4982190: wanted 24, got 0",,,,,,,0,,"xlog.c",4496,
2026-10-02 22:27:16.997722 PDT,,,p2374196,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","redo done at 0/4982160 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s",,,,,,,0,,"xlog.c",8033,
2026-10-02 22:27:16.997985 PDT,,,p2374196,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","end of transaction log location is 0/4982190",,,,,,,0,,"xlog.c",8122,
2026-10-02 22:27:16.998173 PDT,,,p2374196,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","checkpoint starting: end-of-recovery immediate",,,,,,,0,,"xlog.c",9315,
2026-10-02 22:27:16.998927 PDT,,,p2374196,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","checkpoint complete: wrote 8 buffers (0.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.001 s; sync files=7, longest=0.001 s, average=0.001 s; distance=20 kB, estimate=20 kB",,,,,,,0,,"xlog.c",9396,
2026-10-02 22:27:16.998939 PDT,,,p2374196,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","latest completed transaction id is 1141 and next transaction id is 1142",,,,,,,0,,"xlog.c",8432,
2026-10-02 22:27:16.999000 PDT,,,p2374196,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","database system is ready",,,,,,,0,,"xlog.c",8564,
2026-10-02 22:27:17.002717 PDT,,,p2371526,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","PostgreSQL 14.4 (Apache Cloudberry 2.1.0-incubating build bdf90c5518f) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11), 64-bit compiled on Apr 26 2026 20:17:15",,,,,,,0,,"postmaster.c",3569,
2026-10-02 22:27:17.002730 PDT,,,p2371526,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","database system is ready to accept connections","PostgreSQL 14.4 (Apache Cloudberry 2.1.0-incubating build bdf90c5518f) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11), 64-bit compiled on Apr 26 2026 20:17:15",,,,,,0,,"postmaster.c",3571,
2026-10-02 22:27:17.008125 PDT,,,p2374208,th-270645056,,,,0,con4,,seg-1,,,,sx1,"LOG","00000","initialized 1 resource queues",,,,,,,0,,"resscheduler.c",267,
2026-10-02 22:27:17.205450 PDT,,,p2371526,th-270645056,,,,0,,,seg-1,,,,,"LOG","00000","background worker ""dtx recovery process"" (PID 2374207) was terminated by signal 11: Segmentation fault",,,,,,,0,,"postmaster.c",4285,

from /var/log/messages
Oct 2 22:27:16 pe1uaer5gpm01 kernel: traps: postgres[2371538] general protection fault ip:e1d282 sp:7ffed4b01c60 error:0 in postgres[6e6000+a69000]
Oct 2 22:27:16 pe1uaer5gpm01 kernel: postgres[2374077]: segfault at 4 ip 0000000000e1d282 sp 00007ffed4b01e10 error 4 in postgres[6e6000+a69000] likely on CPU 28 (core 4, socket 0)
Oct 2 22:27:16 pe1uaer5gpm01 kernel: Code: 84 00 00 00 01 ed 49 c7 c5 1b c0 66 01 48 63 cd 48 8d 04 cd 00 00 00 00 48 29 c8 41 80 7d 00 00 4c 8d 24 c2 75 32 49 8b 04 24 <80> 78 04 70 74 18 83 c5 01 48 8b 13 48 63 ed 48 8d 04 ed 00 00 00

Coredump files
-rw------- 1 gpadmin gpadmin 218914816 Oct 2 22:27 core.postgres.2371538.1791005236.11.500.500
-rw------- 1 gpadmin gpadmin 206233600 Oct 2 22:27 core.postgres.2374077.1791005236.11.500.500
-rw------- 1 gpadmin gpadmin 206176256 Oct 2 22:27 core.postgres.2374207.1791005237.11.500.500
-rw------- 1 gpadmin gpadmin 206172160 Oct 2 22:27 core.postgres.2374232.1791005237.11.500.500
-rw------- 1 gpadmin gpadmin 206176256 Oct 2 22:27 core.postgres.2374241.1791005237.11.500.500
-rw------- 1 gpadmin gpadmin 206176256 Oct 2 22:27 core.postgres.2374250.1791005237.11.500.500
-rw------- 1 gpadmin gpadmin 206176256 Oct 2 22:27 core.postgres.2374260.1791005237.11.500.500
-rw------- 1 gpadmin gpadmin 206180352 Oct 2 22:27 core.postgres.2374272.1791005238.11.500.500

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct.