I can understand wanting to qualify a potential buyer, so you don't waste your time with 'tire kickers'. But, to an extent, you're also taking advantage of them. You should consider offering refunds if:
1. The domain owner doesn't respond. You wasted a couple minutes with a phone call and email.
2. The domain isn't for sale. Again, a couple minutes.
3. The domain owner wants a totally unrealistic price for the domain. Maybe a few minutes more to try to talk some sense into them. But still...
Also, keep in mind, real estate agents don't ask for a fee up front and they do a lot more work showing properties...