Henry Caruthers was known as "Grandpap" by his children and grandchildren. As slaves, he and his family came to Texas from Rutherford County in Tennessee in 1847. The family at this time included his wife Malinda and two sons, Calvin and Jeffery.
They settled on the farm of Joe Lawrence, he was overseer of the Caruthers family. The farm was located on the McCandless land, in Navarro County, Texas .