In India, cannabis is illegal and even if you are growing for your personal consumption, it is illegal. So if anyone watches your plant, they might report to the concerned authority and legal action would be taken against you. Cannabis use in Indian society is not acceptable and we are forced to stay away from cannabis and its users. So, the Indian mentality has developed around this thought and sub consciously we stay away from cannabis. And if someone smells the plant, they search for it and try to destroy it (Or its user).
So, its the perception around the cannabis that disturb the people around rather than its actual smell.