The protesters were dressed in the red colors of the main opposition party – Movement for Democratic Change (MDC) – and chanting songs to denounce the 92-year-old president, who recently said he had no plans to step down.
Some marchers carried banners reading “Old clueless Mugabe must go.”
“Where are our 2.2m jobs?” read one placard, in reference to Mugabe’s 2013 election pledge to create jobs.
“We are not demanding an overthrow of the government, we are demanding a dignified exit for a tired Mugabe,” opposition leader Morgan Tsvangirai said.
Zimbabwe is battling a worsening economic decline, highlighted by company closures and growing unemployment.
“Mugabe has no solution to the crisis. We are here to tell Mugabe and his regime that you have failed,” Tsvangirai told reporters.
“This is about jobs and improving the economy, which is in dire straits,” he said.
Mugabe, who has led Zimbabwe since the country was formed in 1980 from the ashes of white-ruled Rhodesia, has previously said he had no plans to hand over power and ruled out grooming his politically ambitious wife, Grace, as his chosen successor.
The Thursday march was peaceful and countrywide demonstrations are planned in the coming weeks.
Opposition leader Morgan Tsvangirai was quoted as saying that it was just the beginning of a wave of protests geared towards forcing Mugabe to resign.