The Wall family asserts that they were held in slavery in Mississippi until 1961.
He worked the fields and milked cows for white families while believing he had no rights as a man. Peonage is a system where one is bound to service for payment of a debt. It was an illegal system that flourished in the rural South after slavery was abolished. Mr. Cain was born into this system believing that he was bound to these people that held him and his relatives captive. Being unable to read and write also stifled any opportunity that may have presented itself to the Mr. Cain because he was unable to decipher anything.
There's a video of a recent Nightline appearance the family made on YouTube. Nightline says that it was not able to confirm the family's story independently but notes that the US Justice Department prosecuted people for keeping slaves well into the 20th century. (via cynical-c)