Elvin Jones was born in Pontiac, Michigan. By age two he said he knew he held a fascination for drums. He would watch the circus marching band parades go by his home as a boy, particularly fascinated by the drummers (sometimes wandering off for miles after the parade). Following his early passion, Elvin joined his high school's black marching band, where he developed his foundation in rudiments. Jones began service in the United States Army in 1946. He was discharged in 1949, and returned home penniless. Jones said he borrowed thirty-five dollars from his sister when he got back to buy his first drumset.

Elvin began his professional career in 1949 with a short-lived gig in Detroit's Grand River Street club. Eventually he went on to play with artists such as Charlie Parker, Miles Davis, and Wardell Gray. In 1955, after a failed audition for the Benny Goodman band, he found work in New York, joining Charles Mingus's band, and releasing a record called J is for Jazz.

In 1960, he joined with the classic John Coltrane Quartet, which also included bassist Jimmy Garrison and pianist McCoy Tyner. Jones and Coltrane often played extended duet passages, both giving and taking energy through their instruments. This band is widely considered to have redefined "swing" (the rhythmic feel of jazz) in much the same way that Louis Armstrong, Charlie Parker, and others did during earlier stages of jazz's development. He stayed with Coltrane until 1966. By that time, Jones was not entirely comfortable with Coltrane's new direction and his polyrhythmic style clashed with the "multidirectional" approach of the group's second drummer, Rashied Ali.

Jones remained active after leaving the John Coltrane group, and led several bands in the late sixties and seventies that are considered highly influential groups. Notable among them was a trio formed with saxophonist and multi-instrumentalist Joe Farrell and (ex-Coltrane) bassist Jimmy Garrison, with whom he recorded the Blue Note album Puttin' It Together. Jones recorded extensively for Blue Note under his own name in the late sixties and early seventies, with groups that featured prominent as well as up and coming greats. The two volume Live at The Lighthouse showcases a 21- and 26-year-old Steve Grossman and Dave Liebman, respectively. Other musicians of note who made significant contributions to Elvin's music during this period were baritone saxophonist Pepper Adams, tenor saxophonists George Coleman and Frank Foster, trumpeter Lee Morgan, bassist Gene Perla, keyboardist Jan Hammer and Jazz - World Music group Oregon.

Elvin Jones' sense of timing, polyrhythms, dynamics, timbre, and legato phrasing - as well as the sheer mass of sound he produced - brought the drumset to the foreground. Jones was touted by Life Magazine as "the world's greatest rhythmic drummer", and his free-flowing style was a major influence on many leading rock drummers, including Mitch Mitchell (whom Jimi Hendrix called "my Elvin Jones") and Ginger Baker. He appeared as the villain Job Cain in the 1971 off-beat Western film Zachariah, in which he performed a drum solo after winning a saloon gunfight. In 1999, Jones worked with Our Lady Peace on their album Happiness...Is Not a Fish That You Can Catch. He was featured playing drums on the song "Stealing Babies", which was also featured on their 2009 compilation album The Very Best of Our Lady Peace.
Jones performed and recorded with his own group, the Elvin Jones Jazz Machine, whose line up changed through the years. Sonny Fortune and Ravi Coltrane, John Coltrane's son, both played saxophone with the Jazz Machine in the early 1990s, appearing together with Jones on In Europe on Enja Records in 1991.

Jones, who taught regularly, often took part in clinics, played in schools, and gave free concerts in prisons. His lessons emphasized music history as well as drumming technique.
Elvin Jones died of heart failure in Englewood, New Jersey on May 18, 2004. He is survived by his first wife Shirley and his second, albeit common-law, wife Keiko (Elvin married Keiko before divorcing Shirley, meaning that legally he and Keiko were not married). Elvin Jones is survived by his son Elvin Nathan Jones of California and daughter Rose-Marie From of Sweden.