If there’s one question that people have when it comes to retirement, it’s how to actually balance their retirement needs with the other goals that they have with their financial life. For example, if you have heavy credit card debt but a nice nest egg, should you rob Peter Nest Egg to pay Paul Credit Card Debt? It’s a tough question, but there are some things that you might want to consider before you actually go through with taking out a loan from your 401K account.

First and foremost, you need to understand that it really is a loan — you will have to pay the money back with interest. That interest does indeed go back into your retirement account in some cases, but either way — it’s still due, and you will need to make sure that you actually pay it. If you don’t pay the loan back, it does become a withdrawal and unless you happen to be 59 1/2, it’s going to cost you 10% penalty plus having to pay income taxes on the withdrawal. It’s a pretty big bite:

Let’s say that you borrow 5,000 dollars, at a nominal interest rate. If you don’t pay back the 5,000, you will not only have penalties from the loan company, but you will also have to immediately take a $500 penalty (10%), plus you will have to roll the $5,000 into your total income — which can add substantial taxes to your bill if you’re not careful — $5,000 can be enough to push you from one tax bracket to another.

One of the reasons why we put money into our retirement account in the first place is to be able to actually save for the future. If you take money out of the account then you will naturally not have that compound interest magic working on that money. over time, this means that you will have a smaller nest egg than other people that actually held tight to their guns and got all of the money that were set out to receive.

Getting a loan from your 401(k) isn’t something that you can just get for free, either — you will have to still pay a fee, called a loan origination fee. This can range, which is why you will need to get a quote from the loan company to figure out how much money you will have to pay in order to get the loan. Now, this money will often be deducted from the total amount that you’re borrowing, so keep that in mind.

From an organization point of view, you also need to make sure that you realize that the money for the monthly payments will be directly taken from your paycheck every month. So you will need to adjust your budget as well — especially if you have things that are automatically withdrawn after your paycheck is deposited.

Now we have to come back to the original problem — should you do it in the name of paying down your credit card debt? Well, it depends on a few things. First and foremost, it depends on the amount of credit card debt that you’re trying to get rid of. If you think that you’re going to try to pay back all of them and they are all high interest cards, then you should definitely do that. On the other hand, if the credit card debt is manageable, it can be smarter to leave the money in the account and find another way to pay the credit card debt off early — like your tax return. It’s much better than blowing your tax return on a vacation that only takes you further from your financial goals.

Overall, this is something that you should definitely keep in mind if you really want to take a loan from your 401(k) to pay back credit cards.