> to the point that to get the equinoxes/solstices back to where they 'should' be 10 days had to be removed with the Gregorian calendar
Note that the equinoxes and solstices are officially supposed to be on the 25th. By the time of Julius Caesar, that had diverged, but the divergence in reality made no impact on the date of the official solstice. The Gregorian calendar could easily have put the solstices back on the 25th, but chose not to.
By the time of Julius Caesar, the solstices were on the 25th, which is the reason why Christmas is celebrated on that day.
The Gregorian calendar has not restored the time of Julius Caesar, but the time of the First Council of Nicaea (325 AD), when the rule about how to compute the date of the Easter was established.
From the time of Julius Caesar to 325 AD, more than 3 days of drift had accumulated, so the Gregorian calendar would have required 13 or 14 days of correction.
When many countries transitioned to the Gregorian calendar much later, they had to add additional correction days to the initial 10-day difference, about 1 day per century.
Note that the equinoxes and solstices are officially supposed to be on the 25th. By the time of Julius Caesar, that had diverged, but the divergence in reality made no impact on the date of the official solstice. The Gregorian calendar could easily have put the solstices back on the 25th, but chose not to.