Random characters sent sometimes (gergo)

It's hard to describe, while typing sometimes the keyboard will completely spaz out and start sending a bunch of random characters, this has happened on multiple machines and over multiple versions of QMK, but always on the gergo keyboard.

If I'm typing slowly it rarely happens, but if I pick up speed or start to hit my layers like while coding, it happens frequently enough to basically make the board unusable.

My layout isn't particularly complicated and can be found here

I originally thought it was due to spacecadet shift but after replacing with regular shift, it's still happening.

• Keyboard: Gergo
• Revision (if applicable): Rev 02: Distortion
• Operating system: Linux (Debian & Arch)
• AVR GCC version: 9.1.0
• QMK Firmware version: 0.6.393
• Any keyboard related software installed? None

I've got an ergodox infinity (2 actually) with QMK and a very similar layout that don't have any issues