The watchdog said newer banks are likely to encounter a number of barriers that make it difficult to compete with established names on the high street.
Although the process of setting up can be relatively straightforward, problems emerge when banks attempt to attract customers and expand their market share, the OFT found.
This is because many customers show loyalty to established brands and are reluctant to switch providers.
Clive Maxwell, the OFT's executive director for goods, services and mergers, said he hopes the study will "contribute to the wider debate on the future of banking".
He added that a healthy level of competition in the retail banking industry is important to "support growth and productivity in the economy".
According to recent figures from the Financial Services Authority, 7,143 complaints were made against banks every day in the first six months of the year.
By Asim Shah