OK, well, here's the situation-- I did cut out a piece of the vinyl and it looks like it will be enough to flatten out the high spot (though I think I will also put a few screws in). But do I now need to smooth over the edges and the old adhesive with floor compound or something so it's a smooth area? Or will the underlayment take care of that?

Underlayment should be able to soak up the thickness of the old vinyl. Since we don't know what underlayment you are using, you can do a quick experiment. Just unroll your underlayment over the affected area and lay some floor down. If flex is not excessive, pick it up and start laying out the floor. If you notice a gap, go to plan B to feather out the depth. You can practice laying floor (I assume it snaps together and comes apart) anywhere you have similar doubts.