"I'm not actually their father, and they should work on their relationship with the person who actually is"