For the people of Ukraine, at least those who are not ethnically Russian, the failure of Washington and Brussels to stop Moscow’s intervention in the Crimean peninsula and several eastern towns is dispiriting. For the West itself, that failure serves to highlight an illusion that has distorted United States and European Union policy towards Russia since the collapse of the Soviet Empire.

That illusion was the belief that, with the end of the Cold War, the West had a vital interest and moral duty to go east. In the Clinton and George W. Bush eras, the United States and its Cold War allies in Western Europe expanded the Atlantic alliance into Eastern Europe, the Balkans, and the Baltics. Add to this Washington’s attempts to deploy US ballistic missiles in the region and offer NATO membership to Georgia and Ukraine, which Russia had long deemed as its sphere of influence. Seen from Moscow’s vantage point, the calls by the EU and Obama administration to topple the democratically elected, albeit corrupt, pro-Russian government in Kiev in February was further proof, as Anatol Lieven shows in this issue, that neither Brussels nor Washington takes into account Russia’s susceptibilities and legitimate security interests.

Some Western pundits on left and right argue that because Russia lost the Cold War, it should simply come to grips with the reality of decline and accept that Western expansion into Russia’s near abroad is paved with benign intentions. But it’s more complicated. For one thing, strategic interests and traditional motives of prestige made the Baltics and Eastern Europe a matter of intense importance to Russia well before Stalin arrived on the scene.

And as foreign policy realists — from Walter Lippmann and Hans Morgenthau to Henry Kissinger and John Mearsheimer — have long argued, a sphere of influence is a key characteristic of a great power. Even a great democracy such as the United States claimed a sphere of influence before it emerged as a genuine great power. Since President James Monroe claimed for the US a sphere of influence in the Caribbean and Central America in the 1820s, the US has intervened in Haiti, Cuba, Nicaragua, Panama, Grenada, and the Dominican Republic.

None of this is extraordinary; it is the way the world has always worked, and notwithstanding John Kerry’s recent claims that the Monroe Doctrine is dead, it is the way great powers still conduct foreign affairs. Imagine how Washington would respond if another great power, say China or Russia, were to intrude militarily into Central America or interfere in the internal political affairs of northern Mexico.

Moreover, the West’s move eastwards has represented a betrayal of the post-Cold War settlement reached between Washington and Moscow. Unfashionable though it is to say so today, the events from 1989 to 1991 had less to do with a triumph of the American mission to redeem the world and more to do with what amounted to a political miracle. The Soviet regime, a brutal totalitarian dictatorship nearly as tyrannical and aggressive as Adolf Hitler’s Nazi dictatorship, voluntarily jettisoned its Warsaw Pact satellite states in late 1989, then acquiesced in the dismantling of its empire in the Baltics and Ukraine in 1991, gave up communism as an ideology, and embraced democracy — all with virtually no violence.

Such dramatic changes were by no means inevitable. When empires collapse, brutality and bloodshed usually coincide. Think of the British departure from Kenya, Malaya, and the Indian subcontinent; or the French from Vietnam and Algeria; or the Belgians from Congo; or the Portuguese from Timor. What occurred in the case of the Soviet Empire’s collapse was very much the exception.

Why? Why didn’t Moscow resist the forces of change and extend its empire’s life by imposing brutal order on the former republics and client states in Eastern Europe? After all, previous springtimes of reform in the Soviet Empire had been followed by a summer of repression: think East Germany in 1953, Hungary in 1956, Czechoslovakia in 1968, and Poland in 1981.

Why did the Soviet Empire die so peacefully?

The answer lies in recognising that the Cold War ended by negotiation to the advantage of both the Soviets and the Americans. For whatever has been said during the past two decades, both Presidents George H.W. Bush and Mikhail Gorbachev reached an agreement in 1990: in exchange for a reunified Germany’s acceptance into NATO, the West would not enlarge its security commitments. In effect, the bargain gave Washington everything it wanted; the quid pro quo was that it would refrain from doing what it had never expressed any intention of doing: that is, take advantage of Moscow by advancing strategically into what historically had been Russia’s near abroad.

Admittedly, there was no formal treaty to codify this agreement. But that is because there already was an implicit understanding between the Cold War protagonists. As Sergei Karaganov, a Russian strategist and future Putin advisor, has explained:

“In 1990 [Russians] were told quite clearly by the West that dissolution of the Warsaw Pact and German unification would not lead to NATO expansion. We did not demand written guarantees because in the euphoric atmosphere of that time it would have seemed almost indecent, like two girlfriends giving written promises not to seduce each other’s husbands.”

As the American Conservative’s Scott McConnell has observed , “the more versatile and powerful girlfriend did indeed seduce, first Poland, Hungary, and the Czech Republic, and has since pushed furthered into nations and regions that are perceived, by Russians, to be literally part of historic Russia.” Or as the Harvard professor Stephen Walt, who visited the United Studies Centre in late April, has remarked : “Intoxicated by its own self proclaimed role as the ‘indispensable nation,’ the US was increasingly willing to extend security guarantees to almost anyone who asked for them.”

This is why Jack Matlock, the US ambassador to Moscow in the late 1980s and early 1990s, believes that “the US has treated Russia as a loser since the end of the Cold War.” And it is why, as Daniel Deudney and John Ikenberry documented in Survival, many Russians think that the United States has reneged on its side of the bargain.

In hindsight, a strong case can be made that President Bush and his senior advisers James Baker, Brent Scowcroft, and Lawrence Eagleburger were right to be cautious about the collapse of the Soviet Union into 15 separate nations, and the prospect of turmoil and chaos that this process could unleash. Far from expressing excitement over the new freedom in Eastern Europe and the Baltics that Cold Warriors from Truman to Reagan had only dreamed of, the Bush realists chose to project calm. Such concerns were especially justified, given that several Soviet states, such as Belarus and the Ukraine, held poorly guarded strategic nuclear weapons from the Soviet arsenal that were aimed at the United States.

When the Berlin Wall collapsed in November 1989, Bush was not in a celebratory mood. “This is a sort of great victory for our side in the East–West battle, but you don’t seem elated,” the CBS journalist Lesley Stahl remarked. To which the President replied: “I am not an emotional kind of guy.” And when Democrats, such as future vice president Al Gore and Senate majority leader George Mitchell berated Bush for not dancing on the Berlin Wall, the President defended his “reserved behaviour” for fear that triumphalism might provoke the Kremlin into cracking down on the revolutionary movements.

Simply put, Bush was unenthusiastic about the prospect of the Soviet Union’s violent breakup. He refused to immediately recognise Lithuania in 1991 and he went to Kiev in August of that year to warn Ukrainians that independence could lead to “suicidal nationalism.” And when the hardliner coup against Gorbachev unfolded during that same month, Bush was adamant: “We’re not going to overexcite the American people or the world … We will conduct our diplomacy in a prudent fashion, not driven by excess, not driven by extreme.”

By Christmas 1991, Moscow allowed the Baltic states to separate from the Soviet Union. Liberals and conservatives, especially neo-conservatives, have since condemned Bush for failing to more enthusiastically embrace national self-determination among the Soviet republics in 1991. (Leading conservative columnist William Safire dubbed Bush’s address in Ukraine as the “Chicken Kiev” speech.) History, it was widely argued, was on America’s side.

But it is likely that Bush’s low-profile use of American diplomatic and economic leverage over the Kremlin constrained Gorbachev’s ability to crack down on nationalist movements in the Baltics. “I did not want to encourage a course of events which might turn violent and get out of hand,” Bush later recalled in his memoirs, written with his national security adviser Brent Scowcroft. In February 1990, Bush warned: “The enemy is unpredictability; the enemy is instability.”

A more belligerent response from Washington, moreover, could have strengthened rather than weakened Moscow hardliners. We’ll never know, but it is a fair bet that by taking into account Russian susceptibilities, Bush helped set the scene for a relatively peaceful transformation of the Soviet empire.

To say again, the peaceful disintegration of an empire is historically abnormal. As the English scholar Martin Wright once observed, “Great power status is lost, as it is won, by violence. A Great Power does not die in its bed.”

hich brings us to the Russian incursion into Ukraine in recent months. Many liberals and conservatives alike insist that although Russia remains a great power, it is a rapidly declining great power. It does not have the capacity or right to insist on a vast sphere of influence.

There is no question that Russia is indeed a declining great power. Demographics are working against it and the economy continues to remain sluggish. But Ukraine matters far more to Russia than it does to the United States or Western Europe. Ethnic Russians number nearly 60 per cent of the population in Crimea and nearly a quarter in greater Ukraine. Russia’s naval base for the Black Sea Fleet is based in Sevastopol, Crimea. And Ukraine is a next-door neighbour that is a conduit for Russian trade.

When push turns to shove, a great power will be assertive when its vital interests are at stake. And if humiliated further and made desperate, Russia could be dangerous in the way that a wounded animal can be dangerous. Remember, it still possesses an enormous arsenal of nuclear weapons. In such circumstances, providing a Russian nationalist such as the widely popular Putin with a cause to exploit is playing with fire.

In the 1990s, many opponents of NATO expansion made precisely these points. One critic was George Kennan, the intellectual architect of the containment doctrine in 1947. Writing in the New York Times precisely half a century later, Kennan warned: “Expanding NATO would be the most fateful error of American policy in the post-Cold War era.” Among other things, NATO enlargement would likely “inflame the nationalistic, anti-Western, and militaristic tendencies in Russian opinion; to have an adverse effect on the development of Russian democracy, to restore the atmosphere of the cold war to East–West relations, and to impel Russian foreign policy in directions decidedly not to our linking.”

Another critic was James Schlesinger, the former CIA director and defence and energy secretary, who died in March. Schlesinger was a Cold Warrior and an unabashed supporter of America’s containment doctrine. But after the collapse of the Soviet Empire, he changed his thinking about the direction of US foreign policy. Writing in The National Interest also in 1997, Schlesinger warned about “a condition of national hubris,” a compulsion “to instruct others on how to behave,” to hector and attempt to give US law extra-territorial reach. America, he feared, was becoming a “universal international nag” that was alienating many friendly nations.

In its editorial tribute to Schlesinger on 29 March, the Wall Street Journal suggested that Schlesinger, with his impeccable credentials as a Cold Warrior, would today be more resolutely opposed to Russia’s incursion into Ukraine’s Crimean peninsula than official Washington. But Schlesinger thought NATO expansion was a “bad idea” in the mid-1990s. And unlike many conservatives and liberals today, he was sensitive to Russian strategic concerns in the “more complex and ambiguous environment of the post–Cold War world.” The following decade he observed that there are numerous Americans “who come out of the Cold War [experience] and kind of miss the excitement.” According to the Washington Post, Schlesinger, quoting Winston Churchill, said victory in the Cold War argued now for “magnanimity.”

Schlesinger was no softie, but he recognised the stupidity of grinding the face of a defeated foe in the dirt. As the United States and several European Union states intend to punish Russia with more economic sanctions and increase strategic, economic and diplomatic support for Ukraine’s unelected interim and anti-Russian government, there is much to be said for the Schlesinger view. George H.W. Bush, among other American realists, would agree.