Seems X11 forwarding is the culprit. With it disabled, everything runs peachy.
However, I still need a solution that allows x11 forwarding.