A group of kandar peddlers selling Malay food used to rest on the site where the Khalid Mosque is now situated. The late owner of the land, Haji Abdul Khalid.bin Haji Mohamed Tyeb a philanthropist and businessman, on the suggestion of this group of peddlers agreed to turn the land into a wakaf (Muslim property dedicated for the pious, religious or charitable purposed) mosque and named it Masjid Khalid.
The mosque was built in 1917 and was managed by the Muslim Trust Fund Association (MTFA) until 1968. The mosque now comes under the administrativeof the Islamic Religious Council Singapore.
The Khalid Mosque has kept its original structure with minor alterations to accommodate its growing congregation. In 1996, major renovation were carried out to restore the building to its present splendour.
The first Imam of the mosques was Almarhum Syeikh Osman bin Syaikh Omar AlGanus, an Arab from Palembang. He was also a famous Qadi (registrar of Muslim marriages) in Singapore at that time. Almarhum Syeikh Osman was also a great cook and the tradition of preparing food at the mosque still lives on today. This has made the Khalid Mosque famous for good food that is available there during occations on the Muslim calendar.