Police said a single shot was fired as Aslan Usoyan, known as Grandpa Khasan, left a restaurant in the capital.
He was taken to hospital, but died on the way. A female bystander was also wounded.
Usoyan, 75, survived two previous assassination attempts in 1998 and 2010, when he was seriously wounded.
Born in Soviet Georgia, Usoyan was considered the most influential criminal in the former Soviet Union. He was first convicted when he was 19.
Russian MP Alexander Khinshtein expressed fear that his killing might trigger gang wars and more trouble.