Smart moving has smart moving problems. It cannot be fixed unless you literally specify a specific item to work with it. See the cough aptly named issue in this fork of SM. U CANT CLAMB LADDERS IN OTHER MODS · Issue #5 · makamys/SmartMoving · GitHub.
The root of the issue is that Smart Moving has a hardcoded list of surfaces that it lets you climb, rather than letting you climb anything that implements
Block#isLadderto return true.