EDINBURGH—Scotland's vote to remain part of the U.K. is a big relief for British Prime Minister David Cameron, who could have faced calls for his job if Scots had opted to leave the U.K. But the fight for the Conservative prime minister is far from over.

Despite victory in the referendum over whether to end the 307-year-old union, his handling of the campaign has fueled unrest within his own party. Critics say that Mr. Cameron cut a bad deal with Scotland's first minister, Alex Salmond, over the referendum's ground rules and then was too complacent during a two-year campaign against the underdog independence camp.

Critics say Mr. Cameron cut a bad deal on the vote's ground rules. Agence France-Presse/Getty Images

"There's a sense that Cameron misjudged this vote, took his eye off the ball and was complacent when the details were first hammered out with Alex Salmond," said Simon Griffiths, a politics lecturer at Goldsmiths, University of London. Even though Mr. Cameron won, his political capital has been damaged, Mr. Griffiths said.

It comes at an important time for the prime minister, who is girding for a U.K. general election in May. His center-right Conservative Party, which failed to win a majority in the last election and was forced to form a coalition with the Liberal Democrats, has been trailing the center-left Labour Party in opinion polls.

He is also under pressure from the small, but increasingly popular anti-EU U.K. Independence Party, thanks to divisions within the Tories over the U.K.'s membership of the European Union. Growing support for Ukip prompted the prime minister to promise a different referendum—this one on whether the U.K. should exit the EU. Mr. Cameron has said if he is elected in May, he will allow the British to vote on that issue by 2017.

For Mr. Salmond, who spearheaded the pro-independence campaign, the outcome is nonetheless a victory of sorts. Although he failed to secure his decadeslong dream of an independent Scotland, he secured promises of significant powers for Scotland's semiautonomous government in Edinburgh. The result could mean much more autonomy for Scotland than U.K. political leaders had ever intended to give.

Mr. Salmond also moved from being a fringe player in U.K. politics to proving himself as a man many observers say possesses tremendous political skill, charisma and persuasiveness.

Mr. Salmond led the Scottish National Party to victory in 2007, and a second, unexpected win in 2011 gave Mr. Salmond an outright majority at the Scottish Parliament. Almost immediately, he announced plans for a referendum on independence—forcing Mr. Cameron's hand.

A woman shows signs of fatigue as she counts ballot cards in Edinburgh, Scotland. AFP/Getty Images

At first, U.K. politicians thought it was a long shot, and for almost two years there was little to suggest otherwise. But in recent weeks, polls showed a surge in support for independence and analysts said the vote too close to call.

Mr. Cameron had urged Scots to look beyond their deep-seated antipathy for party and pleaded with them not to use the vote to give the Tories a kick. The warnings about the economic risks of independence that he and the other leaders of the main U.K. political parties made appear to have outweighed the unpopularity of the painful austerity policies his coalition government has put in place.

The close result underscores a deeper problem for Mr. Cameron and the leaders of the other main political parties. Mr. Salmond appears to have tapped into a vein of frustration among some members of the public, who feel their concerns are ignored by the political establishment. It is a similar discontent that appears to be contributing to the gains by Ukip leader Nigel Farage.

Hugh Andrew, a 52-year-old owner of a Scottish publishing company, said he felt insulted that U.K. politicians didn't seem to take the referendum seriously until the last minute. "Only in the last few days have you seen the real passion for keeping Scotland in the U.K," Mr. Andrew said. "They thought it would be a shoo-in."

Mr. Cameron now also faces many months of difficult negotiations. As part of efforts to woo voters in the final weeks of referendum campaigning, he and the leaders of the other three parties offered greater transfer of powers from Westminster to Scotland. To bolster their message, both Mr. Cameron and Labour Party leader Ed Miliband said in the run-up to the referendum that voting no to independence wasn't voting no to change.

But the parties have different views about which powers should be transferred, so the three leaders will have to hash out what form the devolution of powers—in areas such as taxes, spending and welfare—will take. Key differences include whether to give Edinburgh full control over the rate of income tax in Scotland or to give it the power to vary it from the U.K. rate by a certain amount.

The vote outcome has wider implications for the British political landscape. Recent opinion polls suggested that Labour voters in Scotland had deserted the pro-U.K. party and joined the independence movement despite efforts by senior Labour figures, including former Prime Minister Gordon Brown, a Scot, in campaigning to shore up support for the union. It was Scottish votes that propelled the Labour Party of former Prime Minister Tony Blair to power in the past.

Write to Jenny Gross at jenny.gross@wsj.com and Nicholas Winning at nick.winning@wsj.com