Okay I went and checked out pose estimation, which delivers a superior result (additionally I might be able to extract which way people are facing which would be amazing for art purposes).

Thanks also for the library you posted, I think internally it also uses YOLOv8.

Now the only remaining questions are:

1. How to do the math to translate the perspective camera image into positions on a floor plan, but I already got an idea where to look as this is known math.

2. How to stabilize results that pop in and out of existence — I guess I will just write my own overlay that has lingering states, at least for entities at the center of the image.

3. How to get all of that running with sufficient performance on my old Laptop with it's RTX2080m, because I won't put my tower into an exhibition space for a month. Either way it doesn't need to be fluid 60fps, maybe even 2 fps would be okay.

4. Into which OSC messages to translate the results, as I am collaborating with a VJ.

All of those questions are sort of managable I think.

Good tip, I read over this topic not thinking it would be of any help for my problem, but now that you mention it... maybe pose estimation works better than tracking. I havw to test this.