While we all know there is a huge psychological component to riding there is no denying that it is also very much a physical skill. Everyone needs to develop a balanced, stable secure seat on a horse. That requires improving your core strength and balance but there is a payoff for that effort. You will fee much more confident in the saddle if you know that you can stick there no matter what happens.
Your posture and balance also have a tremendous influence on your horse's physical, mental and emotional balance. If you slump like a lump your horse will think and move in the same way. One of my biggest challenges as an instructor is convincing people that they do need to develop a certain level of fitness to ride well. If you have a calm, easy going, couch potato horse that you ride at a walk once a week then maybe fitness is not as big an issue. If, however, you want to ride more enthusiastically, if you want to trot or canter well, if you are riding a younger, more energetic horse then you do need to be fit.
I am going to recommend two excellent sources to go to for help in developing a "rider's" body.
Almost everyone in the riding world has heard of Sally Swift and Centered Riding. Mary Wanless is not quite as well know. Many people regard their two approaches as being quite different but I find that their methods mesh well together. Both focus on core strength and stability, development of a balanced seat, effective breathing for reduction of tension and most importantly that mind-body connection.
The great thing about both of these systems is that many of the exercises can be learned and practiced on the ground. We all know how difficult it is to learn new skills and techniques in the saddle when so much is happening at once and you are also dealing with another brain.
There are two Centered Riding books: Centered Riding 1 and Centered Riding 2. I prefer the second book that contains more exercises to practice both on the ground and in the saddle.
Mary Wanless has several books out and some of them are quite in-depth. The book I like the best is Ride With Your Mind Essentials. This is a terrific book and it is simpler and more basic than some of her others. Each chapter is a new lesson with many great exercises and tips as well as some very helpful trouble-shooting sections. This is one of my favourite books and I have taught all of the lessons to my students.
Riding is about developing a partnership with your horse. In a balanced partnership both parties should contribute more or less equally. If you expect your horse to do all the work while you park on his back you will end up with a resentful partner. If you want him to carry you willingly to the best of his ability then you must physically hold up your end of the relationship also.