It is good to hear that we aren't the only ones that's been bothered by this. I think this all depends on the culture specifically in the context to which the word madam is often associated to. In our country, addressing women as madam is a sign of respect; however, it does have the connotation that you are referring to someone older that you. Personally, I prefer to be addressed on a first-name basis, especially when I am younger or the same age as the person I'm talking to. This does not only give a comfortable vibe but also makes both of us feel casual with each other.
Other than feeling old, the thought of being called ma'am or madam kinda irks me, as it sounds like I'm a boss or someone who holds some high position. But I guess this also has to do with being old, as typically, older people tend to have the higher position in a company/organization.