Grout should not crumble! It might be that there is no backerboard under it, in which case it is only going to get worse.
You might be able to check what is under it by gently prying up one of the tiles. Use a flexible putty knife or something similar. You stand a risk of breaking a tile; do have any extras?