- Spent early years on a farm in Beaufort, SC, where his father was a farm worker and moonshiner.
- Spent three years an amateur boxer with only one defeat.
- Made 1964 U.S. Olympic team as an alternate, but when a fighter was injured, he stepped in and won the only U.S. gold medal.
- Turned pro in 1965.
- Became heavyweight champion by defeating Jimmy Ellis in 1970 and was dethroned by George Foreman in 1973.
- Beat Muhammad Ali in 1971 in the Fight of the Century, the first of their three meetings. Ali would go on to win in 1974, and in the Thrilla in Manila in 1975.
- Was inducted into the U.S. Olympic Hall of Fame in 1989, and the International Boxing Hall of Fame in 1990.
- Owned a gym in Philadelphia, and trained son Marvis—a heavyweight contender.
- Toured with a band as Smokin' Joe Frazier and the Knockouts.
- Released an autobiography in 1996: Smokin' Joe: The Autobiography.
- Was diagnosed with liver cancer in September 2011.