"His comments regarding Jaya Bachhanji are improper and unacceptable", Swaraj said on Twitter.
"Mr Agrawal joins BJP with his team".
"I have been compared with those who dance and work in films", Agrawal said, a comment that the BJP was quick to reject. He made the controversial remark in that context.
He described his joining the BJP as "homecoming" and said he would unconditionally do whatever the party wanted him to.
With its reduced strength in the Uttar Pradesh assembly following assembly elections a year ago, the Samajwadi Party is in a position to win only one Rajya Sabha seat from the state.
Sources within the BJP said that not everyone was pleased with Agrawal's induction into the party primarily because of his "insulting" comments for Prime Minister Narendra Modi. Also joining the BJP is his son and an SP MLA, Nitin Agrawal.
"Naresh Agrawal ji felt he could serve the country and his state better if he joined the BJP", Goyal added.
Talking to reporters in the presence of Mr Goyal and other BJP leaders like Gaurav Bhatia and Sambit Patra, Mr Agrawal sounded his displeasure on being denied ticket for Rajya Sabha by the Samajwadi Party leadership. The Samajwadi Party is going downhill.
Agrawal, who was in the Congress earlier, is not new to controversy.