He was born James Joe Brown, Jr. on May 3rd, 1933 in Barnwell, SC. His father, Joe Garner Brown was black and American Indian. His mother, Susie Behlings Brown was black and asian. James was a stillborn baby delivered by his aunts. His Aunt Minnie did not give up after he was born, blowing into his lungs until he finally started crying. His parents were very poor. His father made his living selling tree tar to a turpentine company. They lived in shacks without plumbing or electricity in the woods near Barnwell. His parents split up when he was four and stayed with his father.

His father was a hard-working man, and did his best to raise James. He would work on farms or in gas stations when there was no turpentine work and would also make home brew. He eventually got James's Aunt Minnie to stay with them to help out. The three of them decided to move across the Savannah River to Augusta so Joe could find a better job. They moved in with his aunt, Handsome Washington, known as Honey. Joe soon left and never lived with James again but was still around.

Honey ran a brothel and sold moonshine. Her grandson, Willie Glenn lived with her. He was a year older than James and they became like brothers. James would steer servicemen to the house, shined shoes, and danced on the street for change. Most of the money he made, he gave to Honey. Around this time, James became interested in music. He learned to play the harmonica, piano, and drums from people he met, and would practice every chance he got. The house was eventually closed down, and he and Aunt Minnie moved into a cottage.

He later met an older kid named Leon Austin who taught him to play the piano with both hands. James started sweeping floors of the Trinity Baptist Church so he could practice on their piano when no one was there. He practiced all types of music, whatever he heard. At eleven, he entered an amateur night contest at the Lenox Theater and won first prize singing "So Long". Besides music, James was into boxing, baseball, and football.

James started getting into trouble around the age of fifteen. He began stealing and joined a gang. He was caught red handed stealing a car battery by the police, but he didn't stop. James and some friends broke into cars one night and the police came looking for him. They caught him and put him in jail. He had turned sixteen in jail before he was brought to trial, charged with four counts of breaking and entering and larceny from an automobile. He was found guilty and sentenced two to four years for each count consecutively, for a total of eight to sixteen years.

In prison, James returned to his music. James tried to make the best of his time in prison. He became close with the warden, Walter Matthews who treated him like a son. He started a gospel quartet that included Johnny Terry who became a member of the Famous Flames. There was a piano in the gym, and James was eventually allowed to play it earning the nickname "Music Box". It was in jail that he heard about Bobby Byrd, and they first met through the prison fence. James wrote a letter to the Parole Board asking for early release. Someone from the Parole Board came to visit him and James told him he wanted to sing gospel if he left. The next day, he was brought to see the warden. He had made a good impression and was to be released the next day, three years and one day after his conviction. There were some conditions: he could not return to Richmond County (where his family lived in Augusta), he had to find work, and he had to find a place to live. He ended up in Toccoa, working at a car dealership and lived with Bobby Byrd's family who helped him. This was the beginning of a long relationship between him and Bobby.