I had a series of walkways laid down using pavers. Benderboard was used as the frame and fine sand was swept inbetween the paver joints.

After the workers hosed down the walkways, I see that more sand should now be swept inbetween the joints.

Question: would there be a problem if I used brick mortar in place of the fine sand to fill in the spaces between the joints?

Some people say that's not a good idea. Why not?