The intersection of State and Pearl Streets was one of early Albany's principal crossroads. During the seventeenth century, it was a Van Rensselaer family property that devolved to Alida Schuyler Van Rensselaer Livingston.

The name commemorates a legendary elm tree that reputedly was planted during the 1730s by a young Philip Livingston in front of his father's house on the northwest corner. In the days before Dutch Elm Disease, that tree grew to be an Albany landmark. This picture (from the vantage point of the old Schuyler house) shows the intersection in 1815 when the elm tree would have been about 90 years old. At that time, the northwest side was known as "Webster's Corner" as the yellow frame building was then the printery and newspaper office of Charles R. and George Webster. The old-style structure on the northeastern corner was the landmark Lydius House - at that time the residence of the reclusive Baltus Lydius. North, on the west side of Pearl Street, was the double townhouse called "Vanderheyden Palace."