Mifflin Gibbs was born on April 17, 1823, in Philadelphia, Pennsylvania, the eldest of four children born to Jonathan and Maria Gibbs. His father, a Methodist minister, died when Mifflin was a child, and his mother worked as a laundress.
Gibbs learned carpentry through an apprenticeship. He read widely and attended debates at the Philadelphia Library Company of Colored Persons. He had a chance to practice his own oratory in the 1840s when Frederick Douglass invited him to help conduct an abolitionist lecture tour.
Journeying to California soon after the gold rush of 1849, he became a successful retail merchant and a leader of the growing black population in San Francisco. He was a founder of the first black newspaper west of the Mississippi River, The Mirror of the Times (1855). He left San Francisco in 1858 to escape growing racial prejudice on the California frontier.