Okay I should mention that at the very least CoFH-Core does no longer cause this Issue with GT6 6.14.00 or later installed, because @OvermindDL1 provided me with a piece of ASM that stops CoFH-Core from lagging on Server start.
But still, the Solution if you get this Issue continues to be:
Step 1: Fork Forge and make your own better Forge
Step 2: Fix that Bug in your better Forge
Step 3: Install said better Forge onto your Client
Step 4: Never have to bother with that Issue ever again
Step 5: share the fixed Forge with us!
Unless you want to host a dedicated Server on your Local Network and join it with your Client, since this Issue only happens in regular Singleplayer.