I'm not sure that's true so much as doing admin and doing EM are shockingly hard to integrate in a way that's successful on both fronts. Mostly this comes down to a mix of the admin side being under-resourced and the expectations of other EM docs that you a) work a full range of shifts and b) are as clinically productive on shift as any of your colleagues. Every other physician led department is going to bend around the physician leader's schedule or have significant admin support from the department itself. Especially in CMGs, an administrative assistant is rare below the level of regional director and most hospitals expect you to be on call 24/7 for administrative issues.
In terms of expectations from the docs, most meetings are going to be weekdays at 7a-9a, or at noon. If you work an overnight, you're trying to keep the contract while having either no sleep or having gotten 3-4 hrs. If you work weekends, you never see your family and end up divorced. If you work a lot of afternoon shifts, you end up pulling 16 hr days routinely. And when you work during the day, everyone in the hospital comes up and wants you to emergently deal with their issue. Nobody tries to pull a surgeon performing a colectomy into a sepsis meeting, but they'll think nothing of dragging you into ad-hoc meeting around blood culture contamination rates when you're trying to put in orders on a crashing patient. This makes it hard to keep pace clinically, both from the time perspective of actually dealing with the issue but also the constant task switching.
If you take a good clinical doc in the community and make them do significant admin, you'll find that a lot of them would end up looking like your current group of medical directors or they quit/step down quickly. Sure there are people doing it for the "cushy" schedule or as a powertrip. Even in the absence of those motivations there are plenty of reasons why EM leaders are looked down upon that have nothing to do with the quality of the person and everything to do with how the job is set up.