Yes, you need to insulate the ducts, all of them that you can. You can get flat duct insulation from a HVAC source. The kind you want has a tough face and is about 1" or more thick. You glue the other side to the ducts. There may be a better adhesive but 3M spray-on adhesive will work.
If you cannot get that insulation, foam sheet insulation will also work. Make sure you use the right adhesive.