He has been backed by two Republicans who were at the White House meeting, but Democratic Senator Dick Durbin said Mr Trump called African countries "shitholes" several times and used "racist" language.
On Friday, AU chief Moussa Faki's spokeswoman Ebba Kalondo had also said Trump's comment "truly flies in the face of accepted behaviour and practice".
"I think you all know that I am not going to opine on what the president may or may not have said".
The UN has also branded as racist, the reported remark branding Haiti, El Salvador and unspecified African nations as "shithole countries". Friday night in Raleigh, African immigrants had plenty to say about the president.
He pointed out that the US accepted 23,000 Haitian immigrants and 172,000 Mexican immigrants in 2017, and immigration is so brisk from El Salvador that one in three Salvadorans now lives in the U.S.
"The so-called bipartisan Daca deal presented yesterday to myself and a group of Republican Senators and Congressmen was a big step backwards", he said referring to the deferred action on childhood arrivals Bill that aims to give protection to "dreamers", undocumented immigrants who arrived to the United States as children.
But Democratic Senator Dick Durbin who was reportedly at the meeting said the United States president used "shitholes" in the context of African countries. There is no other word you can use but "racist".
Trump's insults sparked the African group of ambassadors to the United Nations to issue an extraordinary statement condemning his "outrageous, racist and xenophobic remarks" and also demand a retraction and apology. Trump allegedly said after being presented with a proposal to restore protections for immigrants from the countries in question.
"Has America become a better country as a result?" He tweeted, "The language used by me at the DACA meeting was tough but this was not the language used".