The X Factor judge sparked backlash last month when he revealed his "only regret" about his time working with 1D.
