Nothing is ever 100% safe. You can use escrow and if the domain you purchased was stolen, yes, you will lose the domain, because the rightful owner did not sell it to you and it's also likely you will NOT get your money back either. Here is a recent story of when this happened. It's stated at the bottom of the story about not getting the money back.

Now each case would be unique via Escrow, but you will NOT get your money back most of the time. At times, you can get your money back, but it can be after a very long period of time.

If you are the Seller, the money will be protected the most. So when you are selling, Escrow is near 100% safe route to go.