Stairwells strewn with debris and walls crumbling slowly to dust, it is the island that New York forgot for 50 years.