The bricks for the house were made in a field on the side of Iron Hill and the six fluted columns of the porch were hand made from trees cut on the property. The house is three stories high with fifteen rooms which includes eight bedrooms and four fireplaces. Michael Keeley managed the farm and lived in the nearby house whose well supplied the water used by the household. The kitchen was located in the basement where cooking was done in a fireplace six feet in width that remains to this date. Also remaining is a small stone bridge in the front yard over the stream, the scene of fighting during the battle of Cooch's Bridge, September 3, 1777. It is shown on a plan made that year by British officers Howe and Clinton. The original plan is in the possession of the William L. Clements Library, Ann Arbor, Michigan.

Eliza Baynard was a descendant of Rev. Thomas Griffith who was among one of the first Welsh settlers in Welsh Tract in 1703. Eliza resided in Baltimore, Maryland prior to buying the 120 acres for $2,150.00; she would spend the summer at Baynard Hall and the winter in Baltimore. When Eliza Baynard died in 1883 the land was purchased by Edwin S. Armstrong, who resided there for forty-nine years. For this reason the property is often referred to as the Armstrong House.

Prior to the construction of Baynard Hall the area was part of a working farm owned by Levi G. Cooch, who purchased the land in 1839 for $1,600.00 from Thomas and Eve McCombs of Pencader Hundred.

Deeds to the property can be traced back to William Davis and David Evans who were granted the 30,000 acre tract called Welsh Tract by William Penn in 1701. Some of the deeds mention a tollhouse in the area of Cooch's Bridge Road and the Christiana and Elkton Turnpike now known as Old Baltimore Pike.

In 1932 James 0. G. Duffy sold the house and land to Miriam Armstrong Weihe. By 1988 the property on which the house was located had been reduced to 50 acres owned by Cannonshire Associates, who then sold the house and 4 acres to the current resident.