If you cut out a section of carpet in front of the entrance to replace with vinyl, you will need to trim out the cut carpet edge with some kind of hold down strip. Generally where it butts up to another type of flooring, you use an aluminum strip that has gripper teeth to catch the underside of the carpet and it has a rolled over edge that is hammered down on top of the carpet to clamp in to the teeth underneath.