They used multiple ways to get in, pickaxing it open was one of them. Especially if you cant get a Ram or Ladders up a Hill.
And the reason I say that it needs to take ages to work is pure consequence of making it harvestable by the Enemy. If the Enemy can just harvest it as fast as the Builder can, the Wall will be open very quickly.
Just run up to a Wall of lets say Minecraft Obsidian with an Efficiency 5 Pickaxe. Sure if you are there you can stop that Person, but what if you aren’t home or if you dont notice it in time? They will just go woop and they are in your Base, within less than a Minute. And this clearly cannot be what you envision it to be, because that is broken as heck.
Building and Breaking things will be substantially easier in a Game than in real Life, which does come with unique Design Challenges.
ARK Survival (or was it Conan Exiles?) has the Multiplayer Issue of people being forced to build a huge pillar with a God Bubble at the top, just to protect their Items while they are offline.
There needs to be a mechanic where Enemies have 1000 times slower Building/Breaking Speed when you claim a Chunk, in order for your Vision to ever work, you are aware of that, right?
Because if it is not biased in ANY way at all, it will lead to the Enemy just running into your Base and wrecking your shit no matter what, and then you will end up with “nobody is allowed to know where my Base is, or else everything gets stolen!”, which is absolutely not fun at all.