Off course it matters. Be it a casual or official meeting, appointment or simply hangouts, being on time or punctual makes you reliable. This means that people can take you by your word and trust you'd be there at any speculated time.
Always being punctual or on time for any given occasion doesn't end with all the afore mentioned, other good qualities can also be associated with a person on the said subject. Responsible is one of them. As we all know, responsibility is all about a person's ability to be accountable, take control and observe proper management. Hence, being punctual can closely be related to time management. If you prove to have this quality over time, the society will see you as responsible.
On the other hand, people who always turn up late get judged instantly by others. They either refer to them as unorganized or irresponsible. Very little trust will be given to such people as they tend to be unreliable.
Drawing from the above, It is of my opinion that being punctual or in time always matters in the society as it can be used to judge a person's personality.