The 43-year-old boxing legend retired from the sport in 2017 after he notched up an undefeated record of 50-0.
And Mayweather has previously taken issue with the idea that Ali should be named ahead of him as the best the sport has ever seen.
Speaking to Fat Joe on Instagram Live, 'Money' said his top three would include himself, Pernell Whitaker would be second and Roberto Duran would rank third.
However, Mayweather then revealed he would pick Larry Holmes ahead of any heavyweight in history.
The future boxing Hall of Famer explained: "We're going to have to go, OK, we'll go 'Money' Mayweather, we got to go Pernell Whitaker, imma go Roberto Duran -- man, you put me in a tight predicament.
"As far as with the heavyweights, if we're going to go heavyweights, I'm going to be honest, I've got to go Larry Holmes."
Fat Joe insisted that he doesn't believe Holmes is better than boxing legends Ali or Mike Tyson.
But Mayweather went on to provide the full "breakdown" behind his decision to pick Holmes for his No 4 spot.