Boy this forum doesn't get much traffic does it. ;)
I'm no expert on this, I only came here to ask a question. But from a physics standpoint, brick is basically clay, or close to ceramic, which is a great insulator, one of the best. So it stands to reason that's why it's cooler. Also just the sheer thickness of the brick, regardless of what material it would be, would be better than the wood alone. Even 3-4" of metal would be better. Even though it's not an insulator per say, it's just the thickness of it between the inside and outside that would work better than the wood.
You don't say if it's double-walled, so if it's not, just put one wood wall on one side (which I assume is already on the outside), and another on the inside against the 2x4's with some insulation in between. The air space will be a great insulator. If you're already doing that, then other than adding brick, some kind of internal wall against the existing wall would help. Like maybe wall-to-wall-to-ceiling cork at least 1" thick.
Just remember that generally, (brick not withstanding because it's porous), the denser the material the better the thermal conductance, which is what you DO NOT want because the air pockets in the porosity of a material add to its insulation properties. So cork, fiberglass, Styrofoam and the like will be good.