Excellent question, Danielle, with a short answer and a long answer (which I won’t come close to covering in full).
The short answer about Gov. Sanford is that to disappear for a week was failing in his duty as governor of South Carolina. (Imagine if during his campaign for governor, he had said, “By the way, I don’t think being a governor is a 24/7 job, and sometimes I’m just going to have to have some private space for, oh, five or six days, and not tell you where I am. You understand, right?” Somehow I doubt that the electorate would have agreed.) So his case is easy.
The harder part of your question, particularly for libertarians like me, is about the division of public and private roles if the public role is being performed competently and diligently. Here, I side with the Founders. Google any Founder’s name and the word “virtue” and you will find lots of quotes. Here are some examples. The common thread is that limited government cannot work without virtue in the public at large and in their leaders in particular. I agree adamantly with that conclusion, and attribute much of the current loss of limited government to the loss of that kind of virtue. In that sense, people in high positions (private as well as public) serve as exemplars—”role models” doesn’t begin to cover the gravity of that function. They don’t have the option of being reprobates, even lovable ones.
So that’s why I don’t cut unvirtuous overlings any slack even if they are competent. Mind you, I don’t want us to pass laws to make it easier to get rid of them. I just want all of us to look upon them with withering scorn. I bet you didn’t know it is possible to be both a libertarian and a grouchy old prude.