It is conceivable, depending on the bonding agent. That is one reason why rigid insulation has its own type of construction adhesive. Make sure the adhesive is labeled for use with rigid insulation.

The rigid insulation adhesive also is better for adhering the pourous surface of the insulation: it even states that one side MUST have a pourous surface (foam or fiberglass) for the bonding to occur. You thus should not use general constructon adhesive for rigid insulation.