What I mean is an .rsync-filter with ‘H Cache/‘ or some lines of patterns to exclude. You’ll need to run with -F every time. On the sending side, a recent tar will accept —-exclude-caches if you can be diligent about creating CACHEDIR.TAG.

For sure, you need to exclude whatever "dotfiles" you don't want copied (or explicitly copy the ones you want), particularly caches and other giant hidden things.