No, you shouldn't be worried. Your soul is what's leaving nothing else. What you have gain in this lifetime will not be taken to the afterlife. You will judge for what you have done on earth and that will declare if you live a life of peace and no pain or forever suffering. You should only worry if you haven't repented and broken a commandment which you haven't repent for. See we all good people but if we don't repent for our wrong doings are Earth, it will have consequences in heaven. Like a person who didn't repent and ask the Lord for forgiveness in his or her lifetime will be shown no mercy, and the Lord will say "I never knew you." He will say that because that person never had a relationship with him and lived recklessly. "Depart from me!" Will be his words and it will so powerful when he says it as well.