Theres no simple answer to that. Its all based on your state law. You need to talk to a land use attorney in your state to find out how the easement is established.
Common to the process, usually, is that the use by the non property owner is not permitted, blatant, and continuous over a period of time. This indicates the property owner does not know what he owns or cant control it.