If the building is left unlocked, and the visitor treats the building with the utmost respect and care that is due someone else is property, then i can see no issues.
You may get an angry response if your surprised by an angry owner, but someone who has shown respect, in the way they look after the building, may get a different response.
There are two beautifully maintained climbers huts in the Upper Eigiau valley, one is always locked, whilst the other always remains open.
In all the years Ive walked these areas, the locked premises with a lot of equipment clearly in view behind glass, is respected and left unvandalised.
Even the open shelter, looks a little scruffy inside, but its never been vandalised by mindless idiots, even though it would be easy to do so, due to its very remote location.
Respect a building, leave it in the same condition when you leave, and you should have few issues.
Leaving all your litter behind and wreck the place, as some people are likely to do, well that's a different matter.