I do think that that's not always the case, especially when doing nothing means that I am taking a break from doing something so stressful and hassle. For instance, after a long day of work, I would just like to go out of the office and stare at the cars passing by. Does that mean I'm doing nothing? Certainly not for I am rather giving myself a treat by trying to regain my peace of mind.
More importantly though, you shall not let other people's opinion control you. And even if you're wasting your time, there's nothing wrong with it. It's your time to waste, not theirs. If we are just going to live our life according to other people's terms, then that will be our ultimate failure.