Are you referring to a chimney for a fireplace or a heating system? The signs would be loose and crumbling clay liner on the interior of the chimney. There wouldn't necessarily be visible clues from the exterior.
Did the sweep tell you he was sure the chimney would need relining even BEFORE looking at it? Did he see any evidence from the ground that the claimed to be a problem?
Would you have any idea what you were looking at?, probably not, which is what these scam artists rely on.
Alfred hit a few points, fireplace or furnace? Clay or metal?
Newer furnaces produce exhaust that is cooler,which has a tendency to give up the moisture in it before exiting the house.This moisture is acidic(byproduct of combustion) and will eat at the morter on clay stacks.This is a problem in chimineys on the outside of houses,generaly not a stack that runs in the middle of the house(insullated from cold air).
Fireplace, do you have "sandy" material directly below the chiminey?
I would call a trusted/recommemded heating and air company,possibly two if the work needed is extensive, to inspect the problem