Francesco Cali, believed to be the boss of the notorious Gambino crime family, was killed on Staten Island late Wednesday, the New York Police Department said.
Cali was rushed to Staten Island University Hospital North and declared dead, the sources said.
No arrests have been made at this time.
The Gambino Family was once among the most powerful criminal organizations in the USA, but federal prosecutions in the 1980s and 1990s sent its top leaders to prison and diminished its reach.
Police said there have been no arrests and an investigation is ongoing.
The Gambino crime family is one of what is widely referred to as "The Five Families" of Italian organized crime operations in New York City and other parts of the United States.
Cali's murder is the first hit on a Mafia boss since John Gotti arranged the assassination of then-Gambino head Paul Castellano in 1985.
Cali, who was born and raised in NY, was the nephew-in-law of John Gambino and the great-nephew of Bonanno mobsters Vito Bonventre and Giovanni Bonventre.
Cali, who served on the family's ruling panel for several years, was elevated to acting boss in 2015, replacing then 68-year-old Domenico Cefalu.
According to a 2008 Advance report, Cali - who also went by "Franky Boy", - was born in Brooklyn and married into the Inzerillo family of Palermo and cultivated close ties with members of the Siderno cartel in Italy.
According to local media, Cali ascended through the ranks, becoming a powerful capo of organised crime before turning 40 - less than 10 years after becoming an inducted member.
The Genovese, Gambino, Lucchese, Colombo and Bonanno mafia families are believed to have controlled organised crime in NY for decades.
Prosecutors claimed that Cali was the Gambinos' ambassador to the Sicilian mobsters and the go-between for the two organisations.