The lender, which is part of Lloyds Banking Group, was found to have not taken proper care over the sale of savings products - many of which were offloaded to older consumers.
According to the Financial Services Authority (FSA) - which served the fine - a high number of the complaints were rejected when they should have been upheld.
It was found that many policies were sold to people who had little knowledge of investment practices.
Tracey McDermott, the FSA's acting director of enforcement and financial crime, said: "This fine reflects the Bank of Scotland's serious failure to treat vulnerable customers fairly."
Lloyds Banking Group was recently among a batch of large British banks - including HSBC, Royal Bank of Scotland, Barclays and Santander UK - to fail to meet their business lending targets under the Project Merlin agreement.
By Claire Archer