You can do it but it will scratch easily. I would take the mirror off. Or have a glass company take it off and get rid of it for you. You might be surprised how easily it comes off, but if it is large, you would have difficulty handling it and disposing of it. Have them scrape off the adhesive; that will be easy for them to do.
Once the mirror is off, you will have to patch the places it was glued on, and that is not difficult. Read the drywall patching topic on my wesbite at www.bobf.info.