Gonzalez spurned another gilt-edged chance 10 minutes later, the Bologna defender heading over despite been unmarked at the back post.
Calvo sent a low strike wide as Costa Rica threatened again before Nemanja Matic and Luka Milivojevic combined to find Milinkovic-Savic, whose overhead kick was saved by Navas with the flag up in any case.
For Serbia, it is Switzerland next before their final group game against World Cup favourites Brazil.
The former Chelsea defender made his 104th worldwide appearance, surpassing the previous record of 103 which he held jointly with former midfielder Dejan Stankovic.
Serbia, who last appeared in the finals in 2010, were eager to show their worth in their first game back on the showpiece event.
Costa Rica started positively with Bryan Ruiz slipping Marcos Urena through on goal in the second minute, but he failed to beat the keeper in a one-on-one.
The Costa Rican bench were not impressed with the Serbian's vigorous efforts on the sideline.
"It was an fantastic goal", Costa Rica defender Bryan Oviedo said of Kolarov's effort.
Serbia's defender Dusko Tosic (C) and Serbia's defender Aleksandar Kolarov (L) celebrate their victory at the end of the Russian Federation 2018 World Cup Group E football match between Costa Rica and Serbia.
Serbia return to the World Cup stage after missing the 2014 Brazil world cup and Euro 2016. "Some of us are repeating, but there are teammates who are playing their first World Cup".
Serbia were nearly punished for their lack of penetration when Francisco Calvo dragged a strike wide as Costa Rica enjoyed a short spell of possession just before the break.
He challenged Navas in the 27th minute from just outside the box, but Navas fell on the ball. Navas also stopped Milinkovic-Savic's bicycle kick in front of goal in the 42nd minute, but it was ruled offside.
"To score, I had it in a corner of my mind, but I especially wanted us to play as a team", said Kolarov.
His experienced side were solid throughout, with 23-year-old Lazio forward Sergej Milinkovic-Savic a constant threat.