Charles Spencer has seriously toyed with the idea of wearing a T-shirt printed with the words 'I hate Stephen Fry'.