Software crashes with an User exception

Bắt đầu với stack đã được giải mã trong cores/esp8266/umm_malloc/umm_malloc.cpp và libraries/ESP8266mDNS/src/LEAmDNS_Structs.cpp, LEAmDNS_Transfer.cpp và LEAmDNS_Control.cpp. Tái hiện crash bằng các thiết lập ESP-12, MQTT TLS, WiFi và PlatformIO được cung cấp, sau đó xác định allocation hoặc poison check nào làm hỏng heap; công việc được coi là hoàn tất khi có nguyên nhân đã được xác nhận và một regression test hoặc case có thể tái hiện.

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: [ESP-12]
• Core Version: [framework-arduinoespressif8266 @ 3.2.0-dev+sha.fb8d6d6]
• Development Env: [Platformio]
• Operating System: [MacOS]

Settings in IDE

• Module: [Generic ESP8266 Module]
• Flash Mode: [qio]
• Flash Size: [4MB]
• lwip Variant: [v1.4|v2 Lower Memory|Higher Bandwidth]
• Reset Method: [ck|nodemcu]
• Flash Frequency: [40Mhz]
• CPU Frequency: [80Mhz]
• Upload Using: [SERIAL]
• Upload Speed: [115200] (serial upload only)

Problem Description

Software crashes with an User exception after connecting to WiFi and MQTT TLS

Reset reason   : External System
connected with @ XXX, channel 1
dhcp client start...
ip:192.168.X.X,mask:255.255.255.0,gw:192.168.X.X
Station connected: 192.168.X.X
otaSetup: Setting up ArduinoOTA
timeSetup: Setting time using SNTP
timeSetup: Current time: Sun Nov 12 09:47:38 2023
mqttSetup: MQTT client id: ESP-XXX
mqttSetup: Attempting connection to server XXX.XXX:8883

No poison after block at: 0x3fff2680, actual data: 0x65 0x0 0xa5 0xa5

User exception (panic/abort/assert)
--------------- CUT HERE FOR EXCEPTION DECODER ---------------

Panic umm_malloc.cpp:751 umm_malloc_core

>>>stack>>>

ctx: sys
sp: 3fffecf0 end: 3fffffb0 offset: 0010
...

But it is not always at the same point in software. Sometimes it also starts the HTTP server and transmits several packages before the crash, like this:

Reset reason   : Software/System restart
connected with @ XXX, channel 1
dhcp client start...
ip:192.168.X.X,mask:255.255.255.0,gw:192.168.X.X
Station connected: 192.168.X.X
otaSetup: Setting up ArduinoOTA
timeSetup: Setting time using SNTP
timeSetup: Current time: Sun Nov 12 09:47:38 2023
mqttSetup: MQTT client id: ESP-XXX
mqttSetup: Attempting connection to server XXX.XXX:8883
mqttSetup: Sucsessfully connected.
webserverSetup: Setting up webserver http://192.168.X.X
mqttConnect: connected using PubSubClient
cisternSwitch: switch cistern OFF
mqttConnect: Current time: Sun Nov 12 09:47:52 2023
mqttCallback: Received cistern_timeout_time 60
URI: /status, Method: GET, #Args: 0
getFreeHeap: 13712, getHeapFragmentation: 3, getMaxFreeBlockSize: 13304
webserverHandleStatus: json size/capacity: 741/1024
...
URI: /status, Method: GET, #Args: 0
getFreeHeap: 11792, getHeapFragmentation: 11, getMaxFreeBlockSize: 10576
webserverHandleStatus: json size/capacity: 741/1024
No poison after block at: 0x3fff3008, actual data: 0x65 0x0 0xa5 0xa5

User exception (panic/abort/assert)
--------------- CUT HERE FOR EXCEPTION DECODER ---------------

Panic umm_malloc.cpp:751 umm_malloc_core

>>>stack>>>

ctx: sys
sp: 3fffe9a0 end: 3fffffb0 offset: 0010
...

I have no idea where to search for the problem. Can someone give me a hint (looking at the dump)? There seems to be enough heap available and fragmentation changes between 3 and 19 in the example above but does not constantly grow.

Another this that's strange to me is that ESP.getResetReason() outputs Software/System restart after that. For me this looks like an Exception and not like a "normal" Software/System restart ...

MCVE Sketch

Will follow, if needed.

Debug Messages

ctx: sys
0x40101160: umm_malloc_core at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:753
0x40100ed8: put_poison at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:41
0x40101064: check_poison_neighbors at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_local.c:59
0x401014a2: umm_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:912
0x4022955e: esp8266::MDNSImplementation::MDNSResponder::stcMDNSServiceTxt::allocKey(unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Structs.cpp:116
0x4010150c: umm_poison_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:173
0x401014a8: umm_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:914
0x4022ada7: esp8266::MDNSImplementation::MDNSResponder::_readRRAnswerTXT(esp8266::MDNSImplementation::MDNSResponder::stcMDNS_RRAnswerTXT&, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:678
0x40100940: malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/heap.cpp:250
0x40224a67: operator new[](unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/abi.cpp:52
0x4022955e: esp8266::MDNSImplementation::MDNSResponder::stcMDNSServiceTxt::allocKey(unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Structs.cpp:116
0x4022958f: esp8266::MDNSImplementation::MDNSResponder::stcMDNSServiceTxt::setKey(char const*, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Structs.cpp:131
0x4022adbd: esp8266::MDNSImplementation::MDNSResponder::_readRRAnswerTXT(esp8266::MDNSImplementation::MDNSResponder::stcMDNS_RRAnswerTXT&, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:679
0x40229e0c: esp8266::MDNSImplementation::MDNSResponder::stcMDNS_RRAnswer::stcMDNS_RRAnswer(esp8266::MDNSImplementation::MDNSResponder::_enuAnswerType, esp8266::MDNSImplementation::MDNSResponder::stcMDNS_RRHeader const&, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Structs.cpp:914
0x3fff03f8: _Pri_3_Stack at nmi-vector.o:?
0x4022b1e9: esp8266::MDNSImplementation::MDNSResponder::_readRRAnswer(esp8266::MDNSImplementation::MDNSResponder::stcMDNS_RRAnswer*&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:519
0x4022007c: Stream::Stream() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/Stream.h:63
(inlined by) HardwareSerial::HardwareSerial(int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/HardwareSerial.cpp:41
0x40228bc4: esp8266::MDNSImplementation::MDNSResponder::_parseResponse(esp8266::MDNSImplementation::MDNSResponder::stcMDNS_MsgHeader const&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Control.cpp:770
0x3fff03f8: _Pri_3_Stack at nmi-vector.o:?
0x3fff03f8: _Pri_3_Stack at nmi-vector.o:?
0x4022b414: esp8266::MDNSImplementation::MDNSResponder::_readMDNSMsgHeader(esp8266::MDNSImplementation::MDNSResponder::stcMDNS_MsgHeader&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:1274 (discriminator 2)
0x40278084: print_number at /home/wchen/gwen/nonos/app/main/eagle_lib.c:99
0x3fff0218: _Pri_3_Stack at nmi-vector.o:?
0x3fff03f8: _Pri_3_Stack at nmi-vector.o:?
0x3fff03f8: _Pri_3_Stack at nmi-vector.o:?
0x40228c99: esp8266::MDNSImplementation::MDNSResponder::_parseMessage() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Control.cpp:135
0x40101014: check_poison_block at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:100
0x40101064: check_poison_neighbors at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_local.c:59
0x40100040: _KernelExceptionVector at ??:?
0x40228db0: esp8266::MDNSImplementation::MDNSResponder::_process(bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Control.cpp:80
0x40228de8: esp8266::MDNSImplementation::MDNSResponder::_callProcess() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Helpers.cpp:161
0x4022f801: std::_Function_handler<void (), std::_Bind<bool (esp8266::MDNSImplementation::MDNSResponder::*(esp8266::MDNSImplementation::MDNSResponder*))()> >::_M_invoke(std::_Any_data const&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:293
0x401000e5: std::function<void ()>::operator()() const at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:623
0x3ffeff70: _Pri_3_Stack at nmi-vector.o:?
0x40219694: UdpContext::_s_recv(void*, udp_pcb*, pbuf*, ip4_addr const*, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/include/UdpContext.h:590
0x3ffe8938: ?? at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:123
0x3ffeff70: _Pri_3_Stack at nmi-vector.o:?
0x3fff0208: _Pri_3_Stack at nmi-vector.o:?
0x40244289: ip4_input at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/lwip2-src/src/core/ipv4/ip4.c:1461
0x3ffeff70: _Pri_3_Stack at nmi-vector.o:?
0x40101470: umm_poison_free_fl at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_local.c:161
0x4027458c: ppRecycleRxPkt at /home/wchen/gwen/nonos/app/pp/pp.c:768
0x3ffeff70: _Pri_3_Stack at nmi-vector.o:?
0x4023b031: ethernet_input_LWIP2 at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/lwip2-src/src/netif/ethernet.c:188
0x4023ae4c: git2glue_err at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/glue-lwip/lwip-git.c:118
(inlined by) esp2glue_ethernet_input at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/glue-lwip/lwip-git.c:494
0x402779ea: ethernet_input at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/glue-esp/lwip-esp.c:365
0x402779fc: ethernet_input at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/glue-esp/lwip-esp.c:374
0x40278737: lwip_if0_task at /home/wchen/gwen/nonos/app/main/eagle_lwip_if.c:56
0x40106c05: call_user_start_local at ??:?
0x40106c0b: call_user_start_local at ??:?
0x40101430: umm_init at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:569
0x4010000d: call_user_start at ??:?
0x401000ab: app_entry_redefinable at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:391
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x40102df1: rcUpdateTxDone at /home/wchen/gwen/nonos/app/pp/trc.c:806
0x40105f2d: ets_timer_handler_dsr at /home/wchen/gwen/nonos/app/main/ets_timer.c:335
0x40105c83: lmacTxDone at /home/wchen/gwen/nonos/app/pp/lmac.c:1730
0x4025866e: br_aes_ct_bitslice_Sbox at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct.c:191
0x4024e643: shift_rows at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_enc.c:45 (discriminator 3)
(inlined by) br_aes_ct_bitslice_encrypt at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_enc.c:105 (discriminator 3)
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x4024e814: shift_rows at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_enc.c:45
(inlined by) br_aes_ct_bitslice_encrypt at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_enc.c:110
0x4024cda8: br_aes_ct_ctr_run at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_ctr.c:81
0x4024e814: shift_rows at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_enc.c:45
(inlined by) br_aes_ct_bitslice_encrypt at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_enc.c:110
0x4024cda8: br_aes_ct_ctr_run at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_ctr.c:81
0x4022b36a: esp8266::MDNSImplementation::MDNSResponder::_udpAppend8(unsigned char) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:1178
0x4022b4b9: esp8266::MDNSImplementation::MDNSResponder::_write8(unsigned char, esp8266::MDNSImplementation::MDNSResponder::stcMDNSSendParameter&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:1311
0x4022b77d: esp8266::MDNSImplementation::MDNSResponder::_writeMDNSServiceDomain(esp8266::MDNSImplementation::MDNSResponder::stcMDNSService const&, bool, bool, esp8266::MDNSImplementation::MDNSResponder::stcMDNSSendParameter&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:1488 (discriminator 3)
0x40101014: check_poison_block at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:100
0x3ffe8938: ?? at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:123
0x40101064: check_poison_neighbors at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_local.c:59
0x4010114f: umm_malloc_core at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:751
0x40100ed8: put_poison at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:41
0x40100f0d: get_poisoned at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:121
0x401014a8: umm_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:914
0x40101014: check_poison_block at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:100
0x3ffe8938: ?? at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:123
0x40101064: check_poison_neighbors at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_local.c:59
0x4010114f: umm_malloc_core at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:751
0x401009ac: sdk3_pvPortMalloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/heap.cpp:360
0x4022b36a: esp8266::MDNSImplementation::MDNSResponder::_udpAppend8(unsigned char) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:1178
0x4010150c: umm_poison_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:173
0x4022b36a: esp8266::MDNSImplementation::MDNSResponder::_udpAppend8(unsigned char) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:1178
0x40275317: ppProcessWaitingQueue at /home/wchen/gwen/nonos/app/pp/pp.c:3057
0x40275366: ppMapTxQueue at /home/wchen/gwen/nonos/app/pp/pp.c:3078
0x40275472: ppMapTxQueue at /home/wchen/gwen/nonos/app/pp/pp.c:3146
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x4025cd07: ieee80211_output_pbuf at /home/wchen/gwen/nonos/app/net80211/ieee80211_output.c:313
0x401029c0: pp_post at /home/wchen/gwen/nonos/app/pp/pp.c:3252
0x40105c9b: lmacRxDone at /home/wchen/gwen/nonos/app/pp/lmac.c:1740
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x4010354b: wDev_IndicateFrame at /home/wchen/gwen/nonos/app/pp/wdev.c:324
0x401029c0: pp_post at /home/wchen/gwen/nonos/app/pp/pp.c:3252
0x40103728: wDev_ProcessRxSucData at /home/wchen/gwen/nonos/app/pp/wdev.c:603
0x3ffeff58: _Pri_3_Stack at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/glue-esp/lwip-esp.c:213
0x40103bfa: wDev_ProcessFiq at /home/wchen/gwen/nonos/app/pp/wdev.c:1187
0x40103bfa: wDev_ProcessFiq at /home/wchen/gwen/nonos/app/pp/wdev.c:1187
0x4010390c: wDev_ProcessFiq at /home/wchen/gwen/nonos/app/pp/wdev.c:1072
0x40105f2d: ets_timer_handler_dsr at /home/wchen/gwen/nonos/app/main/ets_timer.c:335
0x401007b8: millis at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_wiring.cpp:165
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x40100795: millis at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_wiring.cpp:158
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x40105f2d: ets_timer_handler_dsr at /home/wchen/gwen/nonos/app/main/ets_timer.c:335
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x40212d63: ESP_WiFiManager_Lite::run() at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/.pio/libdeps/upstream_develop_flash_serial_debug/ESP_WiFiManager_Lite/src/ESP_WiFiManager_Lite.h:863
0x40212fb9: ESP_WiFiManager_Lite::isConfigMode() at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/.pio/libdeps/upstream_develop_flash_serial_debug/ESP_WiFiManager_Lite/src/ESP_WiFiManager_Lite.h:1196
(inlined by) loop at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:953
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:96
0x40215fd1: WiFiClient::available() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClient.cpp:254
0x40215fd1: WiFiClient::available() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClient.cpp:254
0x402181c5: BearSSL::WiFiClientSecureCtx::_run_until(unsigned int, bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:598
0x3ffef600: _Pri_3_Stack at nmi-vector.o:?
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x3ffef664: _Pri_3_Stack at nmi-vector.o:?
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x3ffef410: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:99
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x40218760: BearSSL::WiFiClientSecureCtx::_pollRecvBuffer() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:426 (discriminator 2)
0x3ffef410: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:99
0x40218868: BearSSL::WiFiClientSecureCtx::available() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:443
0x4022dd8a: BearSSL::WiFiClientSecure::available() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.h:277
0x3ffef410: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:99
0x40213f14: PubSubClient::loop() at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/.pio/libdeps/upstream_develop_flash_serial_debug/PubSubClient/src/PubSubClient.cpp:387
0x40230420: base64_encode_value at ??:?
0x40228dc0: esp8266::MDNSImplementation::MDNSResponder::_process(bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Control.cpp:88 (discriminator 1)
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:96
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:96
0x40212fd1: loop at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:957
0x40208dd0: std::_Function_base::_Function_base() at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:240
(inlined by) std::function<void ()>::function(std::function<void ()>&&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:357
(inlined by) Ticker::attach_ms_scheduled(unsigned int, std::function<void ()>)::{lambda()#1}::function({lambda()#1}&&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/Ticker/src/Ticker.h:81
(inlined by) std::_Function_base::_Base_manager<Ticker::attach_ms_scheduled(unsigned int, std::function<void ()>)::{lambda()#1}>::_M_init_functor(std::_Any_data&, {lambda()#1}&&, std::integral_constant<bool, false>) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:237
(inlined by) std::_Function_base::_Base_manager<Ticker::attach_ms_scheduled(unsigned int, std::function<void ()>)::{lambda()#1}>::_M_init_functor(std::_Any_data&, {lambda()#1}&&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:208
(inlined by) std::function<void ()>::function<Ticker::attach_ms_scheduled(unsigned int, std::function<void ()>)::{lambda()#1}, void, void>(Ticker::attach_ms_scheduled(unsigned int, std::function<void ()>)::{lambda()#1}) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:609
0x40105f2d: ets_timer_handler_dsr at /home/wchen/gwen/nonos/app/main/ets_timer.c:335
0x3ffef650: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:95
0x402097c5: esp8266webserver::ESP8266WebServerTemplate<WiFiServer>::begin() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WebServer/src/ESP8266WebServer-impl.h:82
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x3ffef380: _Pri_3_Stack at nmi-vector.o:?
0x40206818: debouncePinTimer_Cb() at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:388
0x4022e294: std::_Function_handler<void (), void (*)()>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:271
0x4022e280: std::_Function_handler<void (), void (*)()>::_M_invoke(std::_Any_data const&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:289
0x4022e28a: std::_Function_handler<void (), void (*)()>::_M_invoke(std::_Any_data const&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:293
0x401000e5: std::function<void ()>::operator()() const at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:623
0x40221911: run_scheduled_functions() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/Schedule.cpp:173
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x40100679: esp_schedule at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:136
0x40224e6a: loop_wrapper() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:269