Around 93 per cent of those who voted stated that they do not support a deal to repay a total of around $5 billion from the state's funds.
About 340,000 people in the UK and Holland had accounts with Icesave, a subsidiary of Landsbanki, at the time of its collapse.
The vote looks likely to affect an International Monetary Fund (IMF) agreement to provide Iceland with $4.6 billion in bailout funding â a deal that is subject to the country repaying its debts to international creditors.
Icelandic officials will now attempt to thrash out an alternative payment deal with their British and Dutch counterparts in an attempt to settle the issue.
British chancellor Alistair Darling has already warned that Iceland will be repaying its debt to the UK for "many, many years" to come.
He told the BBC that it was not fair to expect a small country such as Iceland to immediately repay all of its debt.
But Mr Darling added: "We've tried to be reasonable.
"The fundamental point for us is that we get our money back, but on the terms and conditions we're prepared to be flexible."
Talks are set to resume this week between the three countries, with the central issue set to be the one of the interest rate level on the repayments.
A referendum on the matter was originally triggered back in January when the country's president, Olafur Grimsson refused to sign the bill agreed by parliament to repay the Icesave investors.
The role of president in Iceland is normally a symbolic one and Mr Grimmson's refusal marked only the second time in the country's history that a president has refused to sign a piece of legislation.
By Claire Archer