They will now face Korea in the last eight on Friday while the defeated Australians, who finished runners up in the group, will play Spain.
The Aussies started the contest the better and at one-point held a seven point lead in the first quarter but the Europeans were in resilient mood and they ended the session just two points behind.
Australia were in the ascendancy again early in the second and regular scoring from top performer Abby Kate Bishop, who recorded 14 points with nine rebounds, saw them hold a 35-28 lead at the midway point.
The second half of the contest belonged to the Serbians though, and by midway through the third they had taken the lead as Jelena Milovanovic, who would finish with 23 points and nine rebounds, began to fire.
The contest was nip and tuck now and going into the final quarter the sides could not be separated at 45-45.
But the early stages of the final session saw the Serbians outscore Australia 12-2 and despite a late fightback from the Aussies they held on for the win.