I put the "driver" in quotes: it is not just about ability (let alone perceived ability), but also about equipment (car braking power, tyre quality in relation to road conditions), attention and driving style. You can be reasonably and objectively above average just by having great tyres and strong brakes paired with average attention (which you can control too) and average style/ability.

I agree with that. However, by necessity speed limits need to be set, and enforced, based on vehicles and drivers that meet the minimum standards we set for roadworthiness.

Speed limits are a terrible proxy for actual risk, but the only reasonable way a government can implement anything tending towards "ffs stop killing each other whilst driving too fast for the conditions".