While demolition of the historic building, which dates back to 1830, is not yet being considered, the town's mayor said it could "ultimately" prove to be the safest option for the site.