As Clay Coffman posted above: "Check here: Apps>Google>Settings>Voice>"Bluetooth audio recording" - turn this off." This resolved the issue for me.
I am experiencing the same issue. Voice Input with GBoard is instant, with SwiftKey it is delayed to the point of not being usable. Something changed around February 2020 (guessing when I first n...