The fine from the NYSE Regulation, the New York Stock Exchange's regulatory arm, is comprised of $35 million payment for disgorgement.
Meanwhile, $5 million will be paid to the state of New Jersey for a "separate regulatory matter arising out of the same conduct" and $10 million will compensate customers who invested in the mutual funds.
The unit incurred the fine after it was found that it had not supervised trading of mutual fund shares and variable annuity mutual fund sub-accounts between 2000 and 2003.
Susan L Merrill, chief of enforcement at the NYSE said: "Member firms that inadequately supervise their businesses run the risk of disgorging profits and paying additional penalties.
"The issuance of internal policies and memoranda is not enough: they must be effectively enforced. Actions must follow words."