The fear of dealing with strangers at the beginning of the business is normal, familiar with it will be fine. What you said about not being able to sell directly to strangers is because you just ran the business, inexperienced, psychological pressure caused by too much. It takes time to get used to it.
But if you really don't want to do business, then change the industry. I think your family's financial situation allows you to run the business first to exercise.
What's the question, is your company a manufacturer or an agent?