I was having this same issue for weeks and it started driving me nuts, after some digging I solved it by upgrading to an openjdk fork that supports shenandoah garbage collector. That thing is an absolute game changer, increased my fps by levels of magnitude also, and (if you are familiar with somnia) my sleeping multiplier went from ~50x to ~200x
got all the info from a reddit thread, if you search “Upgrade your Java to one that has the Shenandoah Garbage collector. I got 100-200 more FPS” and you should find it
get the jdk8 fork from redhat and enable shenandoah it really is a game changer.