The short answer is whenever the terms of the contract are breached by the purchaser or clearly defined deadlines have expired. The beautiful thing about real estate contracts is that everyone has the same opportunity to be in the know. Dates are clearly defined. Terms are plainly laid out. The best way to protect your earnest money is to keep track of your deadlines and fulfill your obligations of the contract.
In order to protect your earnest money, you must be aware of all the ways in which it is at risk. Most buyers understand that they are free to back out of a purchase and sale agreement any time during the due-diligence period and receive a full refund of earnest money. Most buyers understand that if they back out of a contract after the due-diligence period, their earnest money is at risk. The one that most buyers aren't aware of is the finance contingency.
The finance contingency is written into the contract by the buyer's agent to protect the buyer's earnest money should the loan get rejected by the lender. This contingency is typically 21-28 days from the binding agreement date (the day the buyer and seller entered into a legally binding contract). This means that the buyer has this number of days to get either an approval or denial from their lender. If this period expires and the buyer has not gotten a yay or nay from their lender, their earnest money is now at risk if they are not able to obtain financing.
The easiest way to avoid getting into a sticky situation is to make sure you are in consistent contact with your lender throughout the process. Make sure they are aware of when your finance contingency ends. When you get to three days before this period ends, ask your lender where your loan stands. If they are unsure or have any doubts about your ability to get the loan approved by the underwriter, ask them to draft up a loan rejection letter. Make sure that your agent has this letter in their possession so when the final day of the finance contingency comes, if you haven't gotten the approval from you lender, your Realtor can send over the rejection letter, terminate the contract, and get your earnest money back in your possession.
Keep in mind, this doesn't always mean that you won't be able to purchase this particular home, and you can always ask the seller for an extension of the finance contingency if your loan officer thinks they can get it done but just needs a little more time.
Protect your earnest money at all times. The number one reason people go to real estate court is over who gets the earnest money when a contract falls apart.