This is the most slippery subject in a long while and don't expect honest opinions as the subject is socially almost unpalatable perhaps unacceptable.
Even more so this days, when obesity is rapidly taking over our once sporty nation.

The basic fitness level of a single handed sailor must be high. Very high.
One might say, common sense should prevail assessing one's own fitness, unfortunately, common sense is far from common.
Let's think of a MOB situation on a crewed boat.
-If one is not able to swim a mile or spend at least twenty minutes in the water unaided, one might be dead. (yes, pfd must be worn, but it is NOT! See VOR Scallywag! )
-If one is unable to pull an injured MOB aboard, he might die.
-If one is not able to climb aboard one's yacht while stationary with the aid of a single rope, he might end up drowning.
Or
-If one is unable to go to the fore deck confidently to handle sails or rigging, all might end up dead.
-If one is not able to climb one's mast with the help of an other person, one might as well stay in the Marina.
So, how fit one has to be? Really?