As ‘ultra wealthy’ seek shelter in quiet town, deeper tensions laid bare