The protesters, mostly women and children, chanted slogans and carried signs Monday denouncing the al-Qaida-linked militant group.
Somali government troops monitored the protest, occasionally firing shots into the air.
Al-Shabab militants have destroyed tombs of venerated Sufi clerics, in what the group calls a campaign to stop Somalis from worshipping at their gravesites.
Monday's demonstration, only the country's second public protest against the militant group, comes just a few days after militants destroyed several tombs in Mogadishu.
Al-Shabab controls much of the Somali capital, as well as parts of southern and central Somalia. The insurgent group practices a very strict form of Islam that forbids idolatry.
Many Somalis practice Sufism, which emphasizes the mystical dimension of Islam, and pay homage at the graves where respected clerics are buried.
Al-Shabab militants have dug up at least seven burial sites in Mogadishu, including the grave of Sheik Moallin Biyamalow. The group had previously exhumed the remains of revered sheiks Aden Dere and Mohyadin Eli.
Al-Shabab and other Islamist insurgent groups have been fighting to overthrow the country's U.N.-backed government and establish an Islamic state in Somalia.