Get a good 1/2 inch drill with a good and sharp 1/2 inch concrete drill bit. Drill four holes at each end of the square. Then if you have the space, drill four more between each hole.
Now with a concrete chissle, chissle out the concrete between the holes. Just about the easiest way. Be careful if you only have a 3/8 inch drill, make sure the bits are new and sharp or you will ruin your drill.