The first library in Townsend was founded in 1858 when 100 citizens gave three dollars each to start a private agricultural library, as was the custom in many towns at that time. In 1873, at the request of the patrons and shareholders, the town took over the running of the library, appropriating $100 annually for its support. Subscribers to the library were also taxed fifty cents a year to help maintain and stock the library. It wasn't until 1882 when this practice was abandoned that it became a truly "free” and public library. However, it wasn't until 1914 that patrons were allowed to browse the collection on their own.

For some years the school board acted as the library committee with the first annual report being made in 1878. The first Board of Trustees was elected in 1893 as mandated by the Commonwealth of Massachusetts.

The library, that consisted mainly of Agricultural Books and Reports from the Agricultural College in Amherst, led a "nomadic existence" in its early years, being housed in private homes, stores and even the Squannicook Fire House No. 1. The plans for the new Memorial Hall included a Town Library room with it's own entrance from the front porch, and that became it's home in 1894. In addition, the West Townsend Reading Room, in West Townsend, and A. C. Josselyn's store in Townsend Harbor were both unofficial distribution centers for library materials from 1910 to 1930. The Reading Room was opened to the town in 1910 by Mrs. Charles S. Homer, and later bequeathed to the town upon her death in 1937.

As a result of bequests from Charles B. Hart and Amanda E. Dwight a Georgian-style brick building was designed by Charles Loring and built on the banks of the Squannicook River on land purchased by the town in 1927. The books were moved from Memorial Hall on February 27, 1929, and the Hart Free Library was opened to the public on March 2, 1929. The official dedication was held on October 30th of that year. Evelyn Warren served as town librarian for 50 years until her retirement in 1945, and remained associated with the library until her death in 1950.

Originally the adult section was in the Fessenden Room and the children's section in the East wing of the main floor of the Hart Free Library. In the 1960s it became apparent that neither section had enough space.   In 1968, with contributions from the newly formed Friends of the Library, the Couple's Club, Stanley Fessenden and other civic-minded people and organizations, the Children's Room was established by Gertrude Hale in the former Historical Society Room on the lower level of the Hart Free Library Building.

Another momentous milestone in the history of the Townsend Public Library occurred in 2007 when Mr. Albert Stone, Chairman of the Sterilite Corporation made an extremely generous gift, to the town, in the name of his company and its employees.  It was the gift of a new, fully furnished library building to be built on Dudley Road at the former site of the Highway Department. As requested, by the donor, the new library building has been named after Richard and Irene Collette, both of whom had served the town in many capacities. Thereafter the building shall be referred to as The Townsend Public Library.