Do you feel old when people call you mam?
In my country we don't usually address women this way but I understand it is a sign of respect when speaking to older women in the US which is why I suppose it could make someone feel old.
In the UK someone might call us madam but it is very formal and not used on a day to day basis. It might be used in a restaurant or occasionally in a shop but I can't remember the last time I was addressed this way and it wouldn't be related to age. A young woman could just as easily be referred to as madam and it wouldn't be offensive. A man might be addressed as Sir but again it is just a sign of respect rather than age related.