You can't weatherstrip the damper (that would be a fire hazard), so build a rustic reclaimed wood fireplace insert to stop drafts. Easy to follow step-by-step tutorial.
