Why wouldn't you also need an inverted X axis to truly mirror your head movements?

That's true. I used to think I was mentally modeling the input as dragging a joystick on the back of the character's head, but it doesn't quite work if you are looking left and right. Now I wonder if it's a lot more subconscious and the "model" explanation is just a rationalization that feels right.

Maybe it's like how some people feel more natural goofy foot on a skateboard/snowboard than the regular way, regardless of their handedness.

I do think a lot of the reasons we believe are just attempts to rationalize what just feels natural to us.

I definitely feel more natural goofy, although I am right handed... but I am also left footed, so I am all kinds of messed up.

The model is dragging the head itself as if it is a joystick. Left and right don't swap, but arrow up maps to forward, which tilts the head down.

The word "mirrors" seems unfortunate here... But yeah, imagine a giant thumb on your head, to make your head look to the sky, the thumb has to pull it backwards.