Yes, definately dont leave it up, or your may find someone claiming your land by adverse possession someday. I would also caution about letting someone else take it down, unless its a real contractor. Even though you didn't put it up, you now own it, and if you allow someone on your land to remove your fence you are subject to some liabilty.