I had been doing the no-brush in the morning for a number of years because I figured there was nothing to brush off, and like you mention enamel can get damaged. Going to the dentist for a deep clean for the first time in a while I was convinced to start doing the AM brush, though. There was some staining building up on the front teeth and brushing twice a day seems to help control it a bit better. Also, the hygienist mentioned how stuff grows on your teeth when you sleep even if you always brush and floss before bed. The bacteria on your teeth makes it easier for stains (from coffee, tobacco, etc) to take hold.
The cleaning process at the dentist must be pretty harsh on the enamel, so I figure I might as well do the extra brushing and maybe they'll take it easier on me. Already, I ask for minimal x-rays and no fluoride if possible.