Poop is associated with dirt because it comes from the body as the food that was not digested. So it is taken as dirty things coming from a human body.
Moreover, poop has a bad oduor that human beings hate. So when you talk about it, people start imagining that bad smell that is associated with it.
Also, I think people find it weird to talk about it since when one goes to the toilet to release it, they do it individually and don't want to be seen by anyone while doing it.