The men were approached from behind by a man who shot them both in the head, a police spokesman said.
Imam Maulama Akonjee, 55, moved to the city from Bangladesh two years ago, reports in New York said.
Police said there were as yet “no indications” the men were targeted because of their faith.
A man holding a gun was seen leaving the scene of the shooting in the Ozone Park area, but no-one has been arrested.
The imam and the second man, who was aged 64, were shot a short walk from the al-Furqan Jame mosque at about 13:50 local time (17:50 GMT) on Saturday.
The second man was named by media as the imam’s assistant at the mosque. He was taken to hospital but died a short time later.
A number of people from the nearby Muslim community have gathered at the scene to demonstrate, saying the shooting was a hate crime, although police said they were still investigating the motive.
“We feel really insecure and unsafe in a moment like this,” Millat Uddin, who lives near the mosque, told CBS New York. “It’s really threatening to us, threatening to our future, threatening to our mobility in our neighbourhood, and we’re looking for the justice.”
Friends of Imam Akonjee told media he had just left the mosque after prayers when he was shot.
“He would not hurt a fly,” his nephew Rahi Majid, told the New York Daily News. “You would watch him come down the street and watch the peace he brings.”