Inconsistent problem running longer tests with SOM on cheyenne with gnu compiler segfault

Start with the SMS_Lm3_D.f19_g16.H_MML_2000_CAM5_BGC.cheyenne_gnu.clm-global_uniform_g16_SOM configuration and the cesm.log segmentation-fault output. Reproduce the long GNU SOM run and compare it with the reported Intel, debug, and short tests; done means the intermittent segfault is resolved or its cause is isolated and documented.

I'm getting an inconsistent problem with SMS_Lm3_D.f19_g16.H_MML_2000_CAM5_BGC.cheyenne_gnu.clm-global_uniform_g16_SOM. It has segfaulted after running for a short period of time, but at an inconsistent time. The first time was 9-steps and the second was after 2105 timesteps.

I also ran debug and short tests without debug for both intel and gnu that all worked. Including a DEBUG 3-month test.

Here's the error from cesm.log

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x2b115b7a494f in ???
#1  0x2b116cc1781d in ???
#2  0x2b115c027af3 in ???
#3  0x2b115ae31bcc in ???
#4  0x2b115ae7078e in ???
#5  0x2b115abd4100 in ???
#6  0xcbe0a9 in ???
#7  0xd4d312 in ???
#8  0xd42762 in ???
#9  0xd31728 in ???
#10  0xca0f76 in ???
#11  0xca18ce in ???
#12  0xc2bdff in ???
#13  0x420f2b in ???
#14  0x40a7aa in ???
#15  0x42014b in ???
#16  0x2b115b7906e4 in ???
#17  0x407268 in ???
        at ../sysdeps/x86_64/start.S:118
#18  0xffffffffffffffff in ???
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 101 with PID 37244 on node r13i7n32 exited on signal 11 (Segmentation fault).

This is running with d1ed49c24fd6b9e0cc55cb60465ca037cc9838ec which I haven't pushed yet, but will. But, I also don't think it's necessarily anything in the new code which is just removing unused code.