Additionally, most of us are more willing to do interview work in our off time. Taking a vacation day to interview is a direct financial investment. Spending a Saturday doing a 4 hour home work is a sacrifice, but doesn't require taking vacation time. Taking an early afternoon off to go talk with 3-4 engineers for 1-2 hours is something that can much more easily fit into ones schedule.

EDIT: I want to provide an anecdote around this. If your salary is $150k, it may cost you roughly $400 in gross pay to interview somewhere. Sure, you could take paid vacation, but vacation time is paid out in cash when you leave so it still costs you. Companies with unlimited vacation are a different story.