If you don't like this person, or don't want to continue to associate with him, consider telling him in a polite way. You can express your own views and feelings, but you also need to respect each other's ideas and decisions. Doing so can not only avoid embarrassment and misunderstanding, but also maintain good personal relations.
Of course, if you think you just want to put aside your emotional considerations for a while and focus on fitness and exercise, it's ok. After all, the gym is a place where you can make new friends and share your life and interests. No matter which way you choose, you should do it according to your actual situation, and don't force yourself to pursue things you don't like or are not suitable for.