In the red-light district of St Louis in 1895, a pimp shot a man dead in an argument over a hat. The ballad telling the story has been recorded by hundreds of bluesmen and jazzers - and even the Clash. It also helped create modern-day rap. Cecil Brown tells the remarkable tale of Stagolee

These are the words sung by the black prisoner Hogman Maxey to the song collector Dr Harry Oster at Angola state penitentiary in Louisiana in 1959.

I first heard the ballad of Stagolee around the same time, sitting under the shade of a tree at the end of the tobacco road in North Carolina. The way my Uncle Lindsey recited the legend, Stagolee was a young god of virility, as impulsive, as vulgar, as daring and as adventurous as the young black field hands wanted him to be. My uncles and their friends recited their rhyming, obscene praise of Stagolee's badness and, at the day's end, gathered in JC Himes's jook joint to dance with girls, drink whiskey and fight. Their nocturnal activities seemed to me to be merely an extension of Stagolee's.

The origins of Stagolee coincide with those of the blues, which sprang up in the 1890s. The first expression of it was as a field holler of former plantation slaves, who carried it with them as they migrated to the work camps along the Mississippi.

Since then, it has taken musical shape as ballad, as blues, as jazz, as epic and as folk song. Its influence can be found in every 20th-century American cultural form, from rock'n'roll to literature to politics to cinema to hip-hop. There are at least 20 jazz recordings, by musicians ranging from Cab Calloway, Jimmy Dorsey and Peggy Lee to Duke Ellington. More than 100 bluesmen, from Champion Jack Dupree and Sonny Terry to Mississippi John Hurt, have recorded it.

During the 1930s and 1940s, the widely respected folklorist John Lomax and his son collected it from prisoners across the south, in the form of a strictly folk protest music; at least a dozen recordings survive in the Library of Congress.

Stagolee - or Stack Lee, or Stagger Lee - has thrived as a soul tune rendered by James Brown, Neil Diamond, Fats Domino and Wilson Pickett. Performers of Stagolee have ranged from levee camp workers to white female "coon-shouters" (white performers who sang as black-face minstrels); from whorehouse pianists to black female blues shouters; from black convicts to Huey Lewis and the News, Bob Dylan and the Grateful Dead; and from 1920s Hawaiian guitarists to 1970s English punks like the Clash, who recorded it in 1979. The earliest recordings, in 1923, were made by two white dance bands, Fred Waring's Pennsylvanians and Frank Westphal and His Orchestra. Australian rocker Nick Cave recorded it in 1996, in Murder Ballads.

Then there are the songs in which Stagolee appears disguised. As Greil Marcus observed, Stagolee was "Muddy Waters's cool and elemental Rollin' Stone; Chuck Berry's Brown-Eyed Handsome Man; Bo Diddley with a tombstone hand and a graveyard mind; Wilson Pickett's Midnight Mover, Mick Jagger's Midnight Rambler... When the civil rights movement got tough, [Staggerlee] took over. And Staggerlee would come roaring back to the screen in the 1970s, as Slaughter, Sweet Sweetback, Superfly."

Though my interest in the legend began in my childhood, it was many decades later, researching a book on the phenomenon, that I discovered the myth's origins, in a murder in 1895 in St Louis, then one of the largest cities in the US. The killer was a man named Lee Shelton.

Shelton was born on March 16 1865, in Texas. Like New York, San Francisco and New Orleans, during the 1890s St Louis had a large red-light district. Most of the ballads say Stagolee was a gambler, but as I dug deeper it appeared that he was also a "maquereau", a French term for pimp often abbreviated to "mack". St Louis, which was founded by the French, still used the term to describe men who were kept by women.

In folk poetry, we find songs praising him:

As a pimp and leader of a group called the Stags, Shelton was a slum hero, reigning in an area called Deep Morgan, one of the few places in the city where blacks and whites could commingle, and where blues, ragtime and "coon songs" had their origins. It was in a bar in this area that Shelton shot and killed Billy Lyons.

According to eyewitness George McFaro, on Christmas night, 1895, around 10 o'clock, Shelton walked into the Curtis saloon, in the heart of Deep Morgan, and asked the bartender: "Who's treating?" In reply, someone pointed out William Lyons. Apparently, the two men drank and laughed together for some time until the conversation turned to politics.

Soon, they began to exchange blows by striking each other's hats. Shelton grabbed Lyons's derby and knocked it out of shape. Lyons said he wanted payment of "six bits" from Stagolee for damaging his derby. Then Lyons grabbed Shelton's hat. At this point, everything changed. The argument turned on Shelton's Stetson, and whether Billy would give it back to him. In an attempt to make him give it back, Shelton pulled his .44 Smith & Wesson revolver from his coat, and hit Lyons on the head with it. Still Lyons would not relinquish the hat. Shelton demanded it again, saying that if Lyons didn't give him his hat immediately, he was going to kill him.

Then Lyons reached into his pocket for a knife and approached Shelton, saying: "You cock-eyed son-of-a-bitch, I'm going to make you kill me." Stagolee backed off and took aim. The 25 people in the saloon flew for the door. Only bartenders and a few others drinking at the bar stayed. Witnesses later testified to the coroner that they then saw Shelton shoot Lyons.

Leslie Stevenson, one of the witnesses, claimed that after he was shot, Billy "staggered against the side of the bar, leaned against the railing, holding the hat in his fingers like that, and it seemed he was getting weak, and he let the hat drop out of his hands. About that time, Stagolee says, 'Give me my hat, nigger' . . . and he picks it up and walks out into the brisk air."

A newspaper account also described Stagolee as walking over to the dying man still holding on to the bar and snatched his hat from Lyons's hand, put it on his head, and walked out "coolly" into the night air.

The murder had serious political consequences. Lyons, it turned out, was a staunch Republican, as were nearly all of St Louis's 25,000 black people. Lyons's stepmother, Marie Brown, owned the famous Bridgewater saloon. Her son-in-law, Henry Bridgewater, was reputed to be the richest black man in St Louis. Lyons belonged to this powerful clan loyal to the Republican party, which had freed the slaves. A new generation, represented by Stagolee, was anxious to vote for the Democrats. Stagolee had gained the support of the Democrats and so was hated by most of the black bourgeoisie, who were represented by Billy Lyons.

In the 1890s in St Louis, black people sought political protection with their right to vote. Both the Republican and the Democratic parties thought they could win if they got the black vote. The majority of black St Louisans voted Republican, but during the Republican convention, in the summer of 1896, many - unhappy that the national Republican party ignored their interest - broke with the party.

This break owed much to the black pimps in St Louis. Under the guise of "sporting" clubs, frequently called the 400 Clubs, pimps, saloon-keepers, and gamblers exerted voting power for the Democratic party. Some saloon-keepers represented the "unofficial" Democratic party.

Many of the figures in the ballad - Judge Murphy, Stagolee's defence lawyer Nat Dryden, Stagolee's wife - were well known figures in the area. Other versions of the ballad make references to historical places and people, like St Louis Chestnut Valley, Lillie Shelton, and bartenders Tom Scott and Frank Boyd. We can assume, therefore, that the hero Stagolee who is the centre of the poem is a reference to the real man Lee Shelton.

Shelton's white lawyer, Dryden, may have been brilliant, but he also was a bohemian with an opium addiction. In the first trial, Dryden got Stagolee off with a hung jury. After two years in the courts, at the retrial in 1897, with a new judge and in the absence of Dryden, Stagolee was sentenced to 25 years in the Jefferson penitentiary. After being released by the Democratic powers that be, he was out for a few years and then returned for pistol-whipping another man. He died in the state prison in 1912, aged 46.

After the murder, the ballad telling of Stagolee's exploits began to spread across the American south and west. A circus performer heard it in the Indian territory in 1913. Early folklorists took an interest in the ballad as early as 1911, when Guy B Johnson published the first version in the prestigious Journal of American Folklore. John Lomax went around the southern states collecting the songs for the Library of Congress during the 1930s. During this time, most black men were either in prison or exploited on farms as sharecroppers. They sang about Stagolee and the Devil. The Devil was the white man.

In 1959, the song Stagger Lee became a number one for the rock'n'roller Lloyd Price, selling a million copies and topping the charts. In the 1960s, the civil rights movement took to Stagolee. At the height of the Black is Beautiful era, James Brown and Wilson Pickett recorded the Stagolee song. Bobby Seale, the leader of the Black Panthers, used it to recruit young black men to the party. I once got the opportunity to ask him why. He replied that Stagolee represented a template for black resistance to whites that just needed to be organised. "Stagolee was a bad nigger off the block and didn't take shit from nobody," he said. "Malcolm X at one time was an illegitimate hustler. Later in life Malcolm X grows to have the most profound political counsciousness... So symbolically, at one time he was Stagolee... To me, Stagolee was the true grassroots."

During this period, Stagolee also took the evolutionary leap that would go on to produce rap music and hip-hop. The connection between the bad man ballads and hip-hop was the form known as the "toast" - a recited story in verse. In telling the Stagolee legend as a toast, the speaker takes on the role of Stagolee. He begins to take on the character of the hero he is singing about. Asserting themselves as bullies and bad men, young black men "perform" Stagolee. The toast became an instrument that allowed them to be powerful and charismatic.

Shedding itself of the musical accompaniment that came with the ballad, Stagolee took his first step to being the basic form that the oral poets, the first hip-hoppers, utilised. During the 1980s, the first rhymers of rap took Stagolee to heightened levels. They used the Stagolee narrative structure as their own personal narratives.

In the development of rap music and hip-hop culture, Stagolee's influence is very clear. It persists in rap in the use of the first-person narrator, the performers' adoption of nicknames, the social drama, the humour, and participation in the commodity culture. From the 1930s to the 1950s, most reciters of Stagolee told the story in the third person. After the rise of the toast tradition in the 1960s, most reciters told the story in the first person. The audience sees through the eyes of the character the rapper creates. The "I" is the bridge between the "I" of the rapper and the "I" of the character.

A reciter of Stagolee associates himself with the hero, but he also makes clear that he is not Stagolee. He can effectively change himself in the eyes of his spectators and listeners. In gangsta rap, the performers are acting out the lives of the criminals in an effort to dispel the criminal from their midst, as a way to get rid of the negative energy.

Stagolee is also present in rap music in the use of cliche: Stagolee is composed of cliche lines that are easy to remember. In rap music, performers found it necessary to use such cliches to keep the rap going.

The final influence that Stagolee has on rap was participation in commodity culture. In the 1890s, the Stetson became a symbol of black male status; in the late 1990s, baggy pants became a signifier of status. As in ear lier generations, ghetto blacks fight against a white appropriation through weird dress. To be able to purchase these commodities, young people in the ghettos resort to hustling, as their parents and grandparents did. They can't afford to believe that a nine-to-five job would solve their problems, because they could never get those jobs.

So gangsta rappers use the lifestyle commodities - cars, clothes, girls - as signifiers of success and wealth. They scrap the old cliche of the ghetto hustler with a slick suit and a truckload of hot goods for the new archetype of the rapper. The term and the concept of the modern-day "mack" are a retrieval of the old cliche of the St Louis mack that Lee Shelton once embodied. And it is not just the mack who is revived, but the women who will do anything for him, including sell their bodies. The girls rappers talk about are whores, or "ho's", just as they were back in the pre-industrial ballads of Stagolee.

Except as appearing in nearly every gangster rap, how does the ballad survive today? What made Stagolee survive for over 100 years? One of the reasons is that performers keep reinventing the song in their own image. Apart from such interpretation from black singers, like Mississippi John Hurt, whites keep reinventing it too.

While he was waiting to record an album, Nick Cave was reading a book on urban black folklore and came across a version of the Stagolee toast. Cave decided to record it for two reasons. He was fascinated, in the first place, by the homosexuality of this particular version. In the toast, Stagolee makes Billy "drop down and slobber on his head". "The final act of brutality, where the great Stagger Lee blows the head off Billy . . . while he is committing fellatio [was] especially attractive," said Cave.

Cave went further, adding lines from another blues ballad. "There's a verse to our version that goes, 'I'm the kind of cocksucker that would crawl over 50 good pussies to get to one fat boy's asshole,' which I heard on an amazing talking blues song by a guy who, in the song, introduces himself as Two-time Slim. I've always thought that was a groovy line so I just threw it in for good measure."

William L Benzon wrote that European-American racism has used African-Americans as a screen on which to project repressed emotions, particularly sex and aggression. We can see this when we look at how white people have used Stagolee. The key to this insight is the concept of projection. "One aspect of this projection," Benzon says, "is that whites are attracted to black music as a means of expressing aspects of themselves they cannot adequately express though music from European roots."

The screen Cave adds to Stagolee tradition tells us more about the culture of the singer than it does of the culture of the song. Stagolee as African-American tradition is the screen that allows the projection to take place. "The reason why we [recorded it] was that there is already a tradition," said Cave. "I like the way the simple, almost naive traditional murder ballad has gradually become a vehicle that can happily accommodate the most twisted acts of deranged machismo. Just like Stag Lee himself, there seems to be no limits to how evil this song can become."

How long will Stagolee get passed on? On a Sunday afternoon late last year, Taj Mahal appeared on stage at Yosi's, a nightclub in Oakland, California. At 60, he is as energetic and solid as he was when I last saw him perform, 20 or so years ago. That afternoon, he had asked that parents come and bring their children. As Taj sang and strummed his big guitar, kids as young as five and as old as 17 were bobbing their heads to the rhythms of the blues.

"Now how many of you kids have heard of bandits," he asked as he stood on the edge of the stage, staring out at the young, white faces. Bandits? Sure. They had heard of bandits. "Okay," Taj said. "And you've heard about gambling? Well, this ballad is about a bandit who gambles! This song is about Stagolee!"

Backstage, sitting in a folding chair, Taj told me how he came across the legend. "The first I heard of Stagolee was from Lloyd Price," he said. "I was a Lloyd Price fan. I was always dancing to him. Then by the 1960s, I kept hearing it on blues anthologies - Mississippi John Hurt, and Furry Lewis's versions. As a child, I'd heard these stories about the bad man - bad man Stagolee - from my mother, who was from the low country in South Carolina. Then there was the other side of my family, my father is from the Caribbean, and from him I heard about 'bad John'. They would say, 'Bad John, stay out he way, man!' "

Taj laughed. This was great fun for him. He was Stagolee. As long as there are living historians like him, Stagolee will never die.

· Cecil Brown's book Stagolee Shot Billy is published on May 29 by Harvard University Press. To order a copy for £19.95 with free UK p&p call the Guardian book service on 0870 066 7979.