There are actually a lot of reasons to be skeptical of experts — the same as why you should be skeptical of anyone. Appeal to authority has a contrapositive: the disappeal to lack of authority. The arbitrary person can be right just as much as the expert can. The whole point of logic and reason is that it’s about facts, not the people.
Critically the problem with your reasoning is the last sentence. No, you do not need to have a medical degree to be right when a doctor is wrong; ask anyone who’s had a medical issue that all doctors agree does not exist.
True, the word was wrong. The point though is that for this logical fallacy, there is an equal and opposite logical fallacy. Saying authority implies truth is just as wrong as saying lack of authority implies falsehood.
Critically the problem with your reasoning is the last sentence. No, you do not need to have a medical degree to be right when a doctor is wrong; ask anyone who’s had a medical issue that all doctors agree does not exist.