Home affairs ministers from the European Union’s 27 countries on Thursday agreed that Croatia should join the Schengen area on January 1.
But Bulgaria and Romania’s accession bid was rejected in a move that EU’s Home Affairs Commissioner Ylva Johansson described as "a disappointment”.
Ms Johansson said she was saddened by divisions over their bid to join the 22-country, passport-free travel zone.
“When we are united in the EU we are so strong, we can achieve so much," she said. "In this moment when it comes to the accession of Bulgaria and Romania we are not united and that makes us very weak."
Romania garnered the most support with 26 votes while Bulgaria received 25, said Ms Johansson.
She did not name the countries that dashed the two countries’ hopes but Austria’s Interior Minister made it clear his nation was against them joining the Schengen.
“I will vote today against the Schengen extension to Romania and Bulgaria,” Gerhard Karner said before the meeting.
“It’s wrong for a system that’s not working in many respects to be expanded further. In Austria, we’ve this year had more than 100,000 illegal border crossings — 75,000 of those are not registered."
Mr Karner called for the vote to be delayed.
Some Dutch politicians had previously voiced concerns about Bulgaria joining the Schengen.
Prime Minister Mark Rutte last week said he wanted assurance that no one could "cross the border with a €50 note", drawing the wrath of the Bulgarian government.
Bulgaria's Interior Minister Ivan Demerdzhiev was quick to reject the "insulting" and "discriminatory" remarks, stressing "the exceptional efforts made to meet the requirements" of European partners.
Decisions on Schengen enlargement require the support of all EU members that fully apply the Schengen rules, as well as the European Parliament.
Romania and Bulgaria joined the EU in 2007 but failed to win a political vote to become Schengen members four years later due to corruption concerns.
Mr Karner was the lone objector to Bulgaria and Romania joining as other home ministers expressed their support and said they hoped for a positive decision ahead of Thursday's meeting.
“We must widen the Schengen area to better control our borders,” said France's Gerald Darmanin.
“Spain supports Bulgaria, Romania and Croatia to come into the Schengen agreement, no doubt about that,” said Spain's Fernando Grande-Marlaska Gomez.
Speaking before the meeting, European Commission Vice President Margaritis Schinas said “the few hesitations” regarding Romania and Bulgaria were “political.”
“They undermine two very simple facts: that we are stronger, not weaker, through Schengen enlargement and secondly that enlarging Schengen means more and better controls," he said.
The three countries “are ready with everything that is necessary to fight smugglers, to prevent illegal crossings to the EU and it’s unfair not to give them the opportunity they have earned”, added Mr Schinas.
Media reports have shone the spotlight on brutal tactics employed by Bulgarian police against migrants, including shootings and pushbacks to stop them from claiming asylum, which is illegal under EU law.
Bulgaria’s Interior Ministry has denied such reports.
The Balkan country lies on a major route for migrants from the Middle East to Europe. Most of them use Bulgaria as a transit corridor on their way westward.
An EU diplomat said an increase of 159 per cent in irregular crossings in the first months of this year was recorded on the Western Balkans route compared to the same period last year.
“There were more than 128,438 border crossings attempts with 22,300 detentions in October alone,” they said, quoting figures from European agency Frontex.
