The building in which the Library is housed was constructed in the mid of 19th century during British rule. It comprises two halls, the Lawrence Hall and the Montgomery Hall. The Lawrence Hall was built in 1863 mainly from the contribution of the European Community of the Punjab and subsequently Viceroy and Governor General of India. The Montgomery Hall was built in 1866 from the subscription of native chiefs of Punjab in honor of Sir Robert Montgomery. Both the halls are joined by a covered corridor. A commodious reading room, leading into the corridor between the two halls, was lately constructed. The building was frequently used as an assembly room for public meetings and staging theatrical and musical performances. Mostly it was a meeting place for the foreign elite to while away their evenings in summer with iced drinks and in winter with log fire. Both the halls were under the charge of the Municipal Committee. The building was known as Old Gymkhana.

On 17th May, 1981 a committee was formed under the chairmanship of then Chief Secretary, Punjab to set up a model library in Bagh-e-Jinnah, Lahore. The renovation work for converting the Old Gymkhana building started immediately thereafter. The governor of the Punjab Lt. Gen. Ghulam Jillani Khan frequently reviewed the arrangements in connection with the establishment of this library. The president of Pakistan Gen. Muhammad Zia-ul-Haq visited Bagh-e-Jinnah to approve the plan on 24th October, 1981. The library was officially inaugurated by the President of Pakistan on 25th December, 1984 and was named as Quaid-e-Azam Library.

The original features of the Old Gymkhana building were restored after extensive labor. The library presents an attractive look after renovation. Library has now capacity for 1000 readers and storage space for more than 300,000 lac books, after the construction of two basements. The floor area of the building including basement is spanned over 70,000 sq. ft.