As with the bombing of Pearl Harbour and the assassination of President Kennedy, all adult Americans know where they were on September 11, 2001. On that Tuesday morning I was five blocks from the White House at the 5th-floor offices of National Affairs in Washington, an office that housed both The National Interest and The Public Interest magazines. The Washington Monument was in its usual place outside my south-west-facing office window as were, of course, the streets below. Once it had become clear that an attack was in progress, national and local media assumed a slightly manic tone. Most private offices reacted by letting their staff go, resulting in gridlocked mayhem throughout the city.

The headless-chicken reaction of the nation's capital to the September 11 attacks disgusted me. Having lived in Israel, it had become second nature for me to assume a stoical mien in times like these, lest one contribute to an enemy's designs. Just as, Eleanor Roosevelt once observed, no one can make a person feel inferior without his or her consent, no one can terrorise you unless you co-operate. I was against co-operating, so I ordered my staff to stay put, do a day's work and go home as usual. Of course we would use the phone to assure relatives and friends that we were safe and we would monitor the news; if necessary, we would adjust to further events. We all stayed until past 5pm, emerging later for the evening commute into a virtual ghost town.

In the past decade, I have often thought of those first few hours after the attacks, and I have come to realise the basic error that US leaders made was to inadvertently co-operate with an enemy too weak to achieve its ends in any other way. To me, September 11 did not "change everything". I thought that, whatever our private worries about the future, the public face of American leadership should radiate optimism and courage, not anger or fear.

Of course, we needed to prevent follow-on attacks. That, it seemed clear to me, meant urgently removing the Taliban regime in Afghanistan that had sheltered and abetted the September 11 plotters. We also eventually needed to take out, in one way or another, those who might be planning more attacks, wherever they might be. The Bush doctrine version 1.0, rolled out in the immediate wake of the attack, which held regimes complicit with terrorism to be equally liable for American retribution, was entirely appropriate. Nor did I object to President Bush terming the situation a war, for that was necessary to tell people what was at stake, to break with the failed policies of the past, and to make available certain prudent presidential legal authorities. Still, despite the need to act, I was sure we should not take ourselves psychological hostage, as the Carter administration had allowed to occur after the US Embassy in Tehran was seized by Iranian fanatics in November 1979. We should not allow the attacks to define or monopolize US foreign policy as a whole.

Alas, that is precisely what the administration did allow. The only senior US leader who seemed to take the approach I thought best was Colin Powell, whose influence had been marginalised in the administration. He did not believe that the terrorist threat was of an existential nature that required the cashiering of American strategic principles, allies or institutions. But other administration principals thought differently, quickly accepting a theory-in-waiting, widely ascribed to so-called neo-conservatives, of why September 11 had happened: a democracy deficit in the Arab-Muslim world had forced frustrated citizens into the mosque, where they had been easy prey for religious charlatans and demagogues. The answer was to open up space for dissent, democratic debate and the social balm supposedly provided by market economics. Then these stultified societies could breathe and develop normally, and would not produce demonic mass murderers like Osama bin Laden.

Thus did fear boomerang, in the way that human emotions predictably do, to encourage a form of hubris fed from the wells of post-Cold War triumph (and triumphalism). The September 11 attacks had the effect of propelling US policy to do more at a time when its capacity to influence events had diminished thanks to the end of Cold War bipolarity and the diffusion of lethal technologies to weak state and non-state actors. It propelled the US to ramp up its metabolism and inflate its definition of vital interests rather than calmly discern distinctions among them. Unrivalled US power, pre-eminently of the military kind, would end the threat by transforming the political cultures of more than two dozen Arab and Muslim-majority countries into liberal democracies. This solution in turn depended on the validity of what was known as democratic peace theory—that democracies do not make war on other democracies—and on cherished Tocquevillian views of the pacific nature of egalitarian democratic societies.

Contrary to what many claim, this theory of the sources of September 11 existed within the administration well before the Iraq war began. It existed within Bush's mind, encouraged by, among others, his speechwriter Michael Gerson and strategic visitors like Natan Sharansky; but it did not have the force of formal policy. The theory emerged into public view when, in February 2003, Bush gave a major speech at the American Enterprise Institute in which all the basic themes of this view found expression. That constituted the Bush doctrine version 3.0, now layered on top of version 2.0, characterised by the pre-emption plank famously inserted into the September 2002 National Security Strategy.

What became known as "the forward strategy for freedom" then found full expression from the bully pulpit in November 2003, with the President's marquee speech at the National Endowment for Democracy (NED). The Strategy was then canonised in his Second Inaugural of January 2005, which Thomas Wolfe aptly dubbed the globalisation of the Monroe Doctrine. The worse things got in Iraq, and the more the WMD rationale for that campaign lost persuasiveness, the higher the rhetorical bar of democracy promotion rose—a classic case of cognitive dissonance at work in what is colloquially called in American poker-speak "doubling down".

The rush to closure over a fearful shock to US security interests, and the hubristic response to it, was part of a longstanding pattern in American foreign policy history. The Bush administration's reactions to September 11 were not the work of any neo-con cabal. Self-avowed neo-conservatives composed a group that was always smaller, more internally diverse and less influential than is often supposed. Rather, neo-cons struck chords very familiar to American history and political culture, chords that even national interest conservatives like Vice- President Cheney and Defence Secretary Rumsfeld could harmonise with. Had there been no neo-cons, the pattern would have asserted itself anyway in some other ideological dialect.

The pattern of which I speak, conceived by the historian Walter A. McDougall, consists of four phases that tend to repeat in cycles. First, there is a shock to the system, usually in the form of a surprise attack: the shot fired at Fort Sumter in April 1861, the sinking of the Maine in Havana Harbour in 1898, the sinking of the Lusitania in 1915, Pearl Harbour in 1941, and September 11 in 2001. In the phase directly after the shock, the leader of the day—Lincoln, McKinley, Wilson, FDR, George W. Bush—vows to resurrect the status quo ante and punish the evildoers. That corresponds to Lincoln's vow to save the Union, Wilson's vow to defend the right of American free passage on the high seas, and Bush's vow to find and punish the perpetrators of the September 11 attacks so that America's minimally acceptable standard of near perfect security could be restored.

But third, in the course of mobilising the national effort to achieve the limited goals set after the shock, the transcendent God-talk begins and the effort soon becomes enmeshed in the sacred narrative of American exceptionalism. This leads to a distension of goals and expectations, to geopolitical amnesia, and to what cognitive psychologists call a dominant strategy that is impervious to negative feedback and logical contradiction.

And so, in the September 11 decade, we chose a war that thoughtlessly destroyed the regional balance against Iranian hegemonism without even stopping to ask about the broader implications of a Shi'a government in Baghdad. One does not, apparently, descend to the smarminess of geopolitical analysis when one is doing the Lord's work. So, too, did we turn what could and should have remained a punitive military operation in Afghanistan into a quixotic, distracted, underfunded nation- and state-building campaign. And so, too, did we conflate all our adversaries into one monolithic demon—typical of eschatological thinking. The administration conflated secular, Ba'athi Iraq with the apocalyptical Muslim fanatics of al-Qaeda, and so went to war against a country uninvolved in 9/11 whose threat to America was not, as is commonly claimed, zero, but which hardly justified, or excused, the haste and threadbare planning with which the war was launched and conducted.

Then, in the fourth phase, overreach leads to setbacks (the Korean War, for example, and the Iraq insurgency) and regrets (like the Vietnam War), ultimately resulting in at least temporary retrenchment ... until the cycle starts all over again. This four-phase model fits the September 11 decade to a tee. The attack itself is of course phase 1; the Bush doctrine version 1.0 represents phase 2; the Second Inaugural signals the full efflorescence of phase 3; and the election of Barack Obama marks the consolidation of phase 4.

It matters in all this, however, whether the ideological vehicle that propels phase 3 forward even remotely reflects or aligns with reality. When it does, as it did during and after World War II, no one pays attention since things tend then to turn out well. In the case of the September 11 decade, unfortunately, it did not. There have been basically two problems with it. First, the "forward strategy" for freedom's ascription of causality for Islamist terrorism is mistaken. Second, even if it were not mistaken, the timetables in which democracy promotion was seen as a solution for mass-casualty terrorism do not even begin to match. The reason is that despite President Bush's assertion that democracy promotion is "the work of generations" and that democracy is about more than elections, that is not the basis upon which the administration actually behaved. It rushed into premature elections in Iraq, Lebanon and the Palestinian territories, with troublesome and still open-ended consequences for Iraq and disastrous ones for Lebanon and Gaza.

After September 11, as Americans searched for analogies that might help them understand the motivations for the attacks, most found themselves with very shallow reservoirs of historical analogies. Indeed, Americans tended almost exclusively to choose Cold War metaphors to explain September 11. Liberal idealists took their characteristic meliorist approach: It was poverty and injustice that motivated the attacks, and American policies that determined the target. There were dozens of calls for a "Marshall Plan for the Middle East", and hundreds of pleas to concentrate more than ever on solving the Arab-Israeli conflict, as if that were somehow a magic bullet that could fix all problems. Conservative idealists, as already noted, took the democracy-promotion approach, arguing that the motivation was not economic but political.

Both were wrong; Islamist radicalism, in truth, is a form of chiliastic violence that has taken many forms in many cultures over the past two millennia, from the Jewish zealots of the First Century of the Common Era, to the 16th-century Peasants' Revolt in Germany, to the 19th-century "ghost dances" of American Indians. But the obvious weaknesses of the meliorist approach encouraged conservative idealists in their conviction that their own view, therefore, must be right. (Manichean-minded Americans have real problems when any potential set of choices exceeds two.)

The administration's rhetoric went even further, however, suggesting that US policy was largely responsible for the debased condition of Arab political cultures. When Bush famously said in November 2003: "Sixty years of Western nations excusing and accommodating the lack of freedom in the Middle East did nothing to make us safe, because in the long run, stability cannot be purchased at the expense of liberty", he argued in essence that it was US policy, not the long incubated political culture of the region, that accounted for Arab autocracy. The Bush White House, in essence, adopted the wrongheaded left-wing side of an old debate over "friendly tyrants" as lesser evils and what to do with and about them, a very strange position for an avowedly conservative administration to take. The President also seemed to be saying, in a locution repeated by Secretary of State Condoleezza Rice in Cairo in June 2005 and many times thereafter, that US Cold War policy in the region was unsuccessful on its own terms, that it did not provide safety and stability.

These claims are nonsensical by any realistic measure. US Cold War policy in the Middle East achieved exactly what it set out to achieve within the broad framework of containment: It kept the Soviets out, the oil flowing to the benefit of the liberal economic order over which the US stood guard, and the region's only democracy, Israel, safe. The record was not perfect, of course, and we certainly should have rethought old habits sooner than we did after the Berlin Wall fell; but it was good enough, as we say, for government work. Certainly, too, it was never in the power of the US government to bring about democracy in the Arab world during the Cold War. Yet the Bush administration's solution for the problem whose origins it misread was just that for the post-Cold War era: deep-rooted reform of the Middle East's sordid collection of autocracies and tyrannies (the major differences between the two were summarily overlooked) and, absent reform from within, the policy strongly implied that pro-democracy regime change would be imposed from without.

The result was almost breathtakingly counterproductive. The more the "forward strategy" bore down on the Middle East, with guns in Iraq and with projects and programs galore practically everywhere else they could gain access, the more effectively local nativists used Western energies jujitsu-like to gain leverage over their domestic adversaries. Rapid economic growth and rapid democratisation, even had they been possible, would not have stabilised Arab societies and made them less likely to spark off political violence against the West; it would have made such violence more likely. We are fortunate, therefore, that the strategy did not "succeed" for any longer than it did.

When the Bush administration campaigned to spread democracy to the Arabs, it never occurred to most of its principals that what they saw as a secular endeavour would be interpreted in the Muslim world through a religious prism, and used accordingly in intra-Islamic civilisational disputes. When Abu Musab al-Zarqawi, the late leader of al-Qaeda in Iraq, tried to persuade Iraqis not to vote because "democracy" was a front, in essence, for Christian evangelism, a slippery slope leading down to apostasy, he spoke a language that resonated in the ears of a great many (though happily not a majority of) Iraqis and other Muslim Arabs.

As it happens, the locals were essentially correct. Americans were speaking a creedal tongue that we thought entirely separate from "religion", a word that does not exist as such in Arabic. After all, we "separate church from state". In truth, American political culture is not as secular as most think: The contemporary American idea of democracy, seen as the pre-eminent symbol of social egalitarianism (something very different from the Founders' view) is an attenuated expression of aspects of Anglo-American Protestant Christian tradition. Our longing to spread it to the Muslims is the 21st-century version of what was, in the 19th century, a much more honest and self-aware missionary movement. We might fool ourselves by pretending that our deepest beliefs can be neatly compartmentalised into what is "political" and what is "religious", but Middle Easterners, who possess no such compartments by dint of a history sans Renaissance or Reformation, know better. Not that theology and ideology are identical, but as creedal systems they are bound to be seen as dramatically less distinct by cultures in which political theology, to use Mark Lilla's apt terminology, has never been vanquished or, in most countries, even seriously challenged.

Looking at US behaviour in the September 11 decade as a manifestation of a secularised political theology explains far more than the standard parsing of the usual-suspects schools of thought: conservative and liberal realists and idealists, Jacksonians and Hamiltonians and all that. Consider for example that when, only days after September 11, Susan Sontag and other members of the professional adversary culture in the United States dared to suggest—in The New Yorker in Sontag's case—that the perpetrators of September 11 were not cowards and that Americans were not innocent victims of terrorism, but rather were suffering just revenge for selfish and abrasive American foreign policies, they were treated exactly as heretics were in the so-called age of religion. They were not engaged or debated but shunned or excoriated. Had it still been in style, they would probably have been burned as witches.

The American penchant for seeing the world, especially the world of foreign and national security policy, in transcendental terms, is not an historical constant. It tends to rise in phase 3 of the cycle, when the God-talk emerges out of post-shock mobilisation. But there is a concurrent trend of more recent vintage that may have made things more acute during the September 11 decade.

Over the past half century, America has become increasingly deculturated. As Robert Putnam put it in his Bowling Alone argument, we have suffered a deep erosion of social capital. The face-to-face glue which enables social interactions to generate and sustain a common understanding about what is and isn't virtuous behaviour—the very heart of what ultimately makes a society prosperous and happy—has been in ever shorter supply.

The implications of de-acculturation for American politics are are manifest. The decline of social trust abets both the polarisation of politics and popular cynicism toward government. It produces a political system in which the chain of connective institutions that link family to neighbourhood to larger community to town or region and ultimately to the national level gets broken, rendering the state both alien and intrusive at the same time as it tries to compensate for a social fabric now rent and tattered. Political parties, particularly those that tend to represent class or ideological structures, flow into the spaces once occupied by a diverse array of social interaction. They become in-group/out-group oriented as well-known psychological dynamics spread the distance between them, leading to an exaggerated perception of how much they actually differ in practical terms. The result is that compromise and horse-trading become more difficult, and the insertion of "culture war" issues into this environment has served only to harden the edges of the us-versus-them distinctions that define it. Identity groups disguised as political parties do not play well together.

The implications of all this for foreign policy are obvious. Presidential judgments necessarily become politicised, and opponents invariably try to criminalise them. Every decision becomes part of the catechism to the loyal, an act of moral enormity to the opposition. That is why the acrid debates over Guantanamo and Abu Ghraib, waterboarding and warrantless wiretaps, the Patriot Act and the reach of wartime executive authority, took on the tones they did. These arguments did not remind one of the civilities of the common law tradition; they sounded more like transcripts from the Spanish Inquisition.

The great sociologist E. Digby Baltzell, the man who coined the term WASP back in the 1950s, once said to me that the greatest tragedy of 20th-century America is that the formidable energies of religion had migrated into politics, to the detriment of both. No wiser comment has ever been made about the trajectory of American politics this past half century, and here lies, I think, the key insight for those trying to comprehend the American September 11 decade at its very core. The decade has not been about what others have done to America; it has been about what Americans have done to themselves.