I am not sure what you have. Can you post a picture of it. Could be somekind of filter, but I am not sure.
Something like these?
"If I opt to replace it with a plug, will shutting off the power shut off the well pump and so prevent water from squirting out the hole? Would water run out of the water tank while I tried to put the plug in?"
Just turning the pump off is probably not enough.
There are lots and lots of different well systems and different setups.
Here are the 3 basic kinds, shallow jet, deep jet, and submersiable pumps.
In the first 2 the pumps might be in the basement or a seperate pump house.
After the tank there is usually a shut off valve then it feeds the house plumbing.
If you just turn off the pump you will still have pressure from the tank. If this is on the other side of the shutoff valve and you close that valve you will won't have pressure, but some water will still come out because of all of the lines, but it will not be under pressure.
Now there are two kinds of tanks used. Most now are bladder tanks and they are typically painted. They will have a schrader valve for charging the bladder. The other is galvanized tanks. They either have air volumne control systems or they can become water logged and need to be drained.
If you have a low producing well you might have low water cutoffs or pump cycling controlls. Also you might have filters or water treatment equipment for things like sulfer or iron in the water. They have special controls. And require certain maintance.
While any plumber can replace the valve or replace it with a plug.
But you really want to get a well person out to go over what you have, what kind of maintance it needs, what can go wrong, where any resets are, what any adjustments are, if the pump needs priming where would you prime it, etc, etc, etc.
It would be an advantage if you find someone that is familar with YOUR well and how much water it produces, what the water level is, etc. So when calling well drillers find out if they have worked on your well/pump.
If you can contact the previous owners ask them who worked on it.
Please let us know what you find out.