Hi LonnythePlubmer and Erik!
Thanks for the thoughts/suggestions. As the problem was not visible in the cold water supply, and it was visible inside the hot water heater (when I drain it), the only places that I could conclude that the problem was at was in the heater and the piping between the cold water pipe and the heater (about 2' of pipe and flexible hose). Therefore, I went ahead and replaced the flexible-hose that supplied the cold wate to the heater. Was unable to remove the hose which took the hot water out with the tools I had.
Thereafter, I sent an email to the LADWP (Los Angeles Dept. of Water and Power), and mentioned the problem, and I got the reply below. It sort of confirms my theory. I did go ahead and replaced the other hose as well.
Time will tell if the problem is solved. Thanks for your help.
The problem you describe may indicate a flex-hose problem. We have found
that some metallic flex-hoses have a black rubber-like lining on the
inside. The chlorine in the water deteriorates this lining over time
causing black particles to show up throughout the home. The black
particles will adhere to the side of sinks, tubs, etc. One
characteristic of this material is that it smears. The problem is more
severe on the hot water side. I would look at your plumbing connections,
if you find flex-hoses which appear metallic on the outside - check the
inside to make sure they're not black. If you have black (rubber) lined
flex-hoses, these should be replaced by "Chloramine resistant"
flex-hoses. If you have other questions or would like to speak with
someone in our office, please call us at (213) 367-3182 between the
hours of 07:30 a.m. and 04:00 p.m., Monday through Friday, except