One of the most significant drawbacks of using your retirement assets is the loss of tax-deferred compound growth for the loan duration. Taking assets out of a retirement account should be a last resort. We discussed withdrawals, and if you do so too early, you pay taxes and penalties.