Camp Nelson was the largest recruiting, mustering, and training center for African American troops (called U.S. Colored Troops) in the Commonwealth of Kentucky and one of the largest in the United States. U.S. Colored Troops trained at Camp Nelson performed garrison duty throughout Kentucky, saw action in both Major General Burbridge's and Major General Stoneman's Southwestern Virginia campaigns, saw action against Confederate raiders at Cynthiana, Kentucky and were involved in the siege of Petersburg, Virginia and the pursuit of the Army of Northern Virginia to Appomattox Court House, Virginia.
Due to the large concentration of families accompanying these troops housing became a major problem. They were expelled from the camp in 1864 amid an uproar from the press. It also lowered morale among the troops and was becoming a national problem. The end result was the passage into law, in February 1865, of the act which freed the wives and the children of the ex-slave enlistees. This act resulted in an increase in the enlistment of enslaved African-Americans in Kentucky and other border states. Assistant Quartermaster Captain Theron E. Hall was appointed superintendent for the refugees and immediately began building barracks to house them in the southwestern part of Camp Nelson.
By June 1865, the refugee camp contained 97 cottages and numerous tents and shacks and provided housing for 3,060 people, primarily women and children. The refugee camp also included a school house, a hospital, a mess hall, a laundry, a lime kiln, teacher's quarters and offices.