What you need to do is determine why the damper flap is not closing completely. Clean out any accumulated grunge, and look for bird's nests, etc. If the flap is closing properly, you should not see daylight, and you will not have drafts.
Good luck!
Andy