Or in other words, scaffolds simply don’t work with smart moving mod. You cannot climb them. I presume smart moving changes the interaction of player with ladders, and most likely the implementation of scaffold climbing doesn’t account for that.
Also, just as a side note, those scaffolds are incredibly gorgeous, their esthetic value is impossible to overpraise.