I think it is more harmful to people who suffer from asthma or other respiratory problems and some are more harmful than others depending on the formulation of the air freshener. Obviously they should be kept away from young children who could swallow the evaporation beads which could cause poisoning but the small quantities we use in the home are not usually hazardous.
When it comes to car air fresheners where they are enclosed in a small space I think it can be more hazardous. Some people have reported having headaches and a feeling of dizziness from the type that hang off mirrors. I use these and although I have not experienced these symptoms, when they are new they are so strong I do find I can taste them. It's never harmed me but it makes me wonder if it could have a negative effect on people who suffer from respiratory problems.