You know that Gigantic Zombie that Minecraft has hidden in the Code? Imagine it would only collide with full Blocks, that is about what you would need to expect in complexity. No real need for Axis Aligned Bounding Boxes (AABB) inside of each individual Voxel.
And since they are so small, most of the time you wont even have to know if the Voxel is sloped or not.