Foreign policy issues were notably absent in the 2000 American election campaign, but they came roaring back within a year. The crisis of September 11, 2001 shocked the country and produced an opportunity for George W. Bush to express a bold new vision of foreign policy. Effective visions combine feasibility with inspiration. Among past presidents, Franklin Roosevelt was good at this; Woodrow Wilson was not. Bush's temperament put him in the Wilsonian camp. He saw himself as a transformational leader, and the Iraq war was a key part of his response to September 11. But by failing to understand the cultural and power context, his responses weakened rather than strengthened the American position in the world. Polls showed a precipitous decline in the attractiveness of the US in Europe, Latin America, and particularly the Muslim world.

In foreign policy, contextual intelligence is the intuitive diagnostic skill that helps leaders align tactics with objectives to create smart strategies in varying situations. Of recent presidents, Ronald Reagan and George H.W. Bush had impressive contextual intelligence. The younger Bush did not, but he was not alone in this failing.

In the discussion below, I will first describe how poorly Americans have understood the context of their power, then survey American hard and soft power, outline the framework of smart power, and conclude with projections for the future of American power.

Academics, pundits, and advisors have often been mistaken about America's position in the world. For example, a little more than two decades ago, the conventional wisdom was that the US was in decline, suffering from "imperial overstretch". A decade later, with the end of the Cold War, the new conventional wisdom was that the world was a unipolar American hegemony. Some neo-conservative pundits drew the conclusion that the US was so powerful that it could decide what it thought was right, and others would have no choice but to follow. Charles Krauthammer celebrated this view as "the new unilateralism" and it heavily influenced the Bush administration even before the shock of the attacks on September 11 produced a new "Bush doctrine" of preventive war and coercive democratisation.

This new unilateralism was based on a profound misunderstanding of the nature of power in world politics. Power is the ability to affect others to get the outcomes one wants. Whether the possession of resources will produce such outcomes depends on the context. In the past, it was assumed that military power dominated most issues, but in today's world, the contexts of power differ greatly on military, economic and transnational issues.

Contextual intelligence starts with an understanding of the strength and limits of American power. On September 11, the US was the only superpower, but preponderance is not empire or hegemony. America could influence but not control other parts of the world. Power distribution at the beginning of the 21st century is like a three-dimensional chess game. The top board of military power is unipolar; but on the middle board of economic relations (where Europe can act as an entity with an economy larger than that of the US), the world is multipolar. On the bottom board of transnational relations (such as climate change, illegal drugs, pandemics, cyber crime and terrorism) power is chaotically distributed. Military power is a small part of the solution in responding to these new threats. They require co-operation among governments and international institutions.

The Bush administration drew analogies between the "war on terrorism" and the Cold War. The president was correct in predicting a long struggle. Most outbreaks of transnational terrorism in the past century took a generation to burn out. But another aspect of the analogy was neglected. Despite numerous errors, Cold War strategy involved a smart combination of hard coercive power and the soft attractive power of ideas. When the Berlin Wall finally collapsed, it was not destroyed by an artillery barrage, but by hammers and bulldozers wielded by those who had lost faith in communism.

There was very little likelihood that the US could ever attract people like Osama bin Laden, but there is enormous diversity of opinion in the Muslim world. Witness Iran whose ruling mullahs see American culture as the Great Satan, but where many in the younger generation want American videos to play in the privacy of their homes. Many Muslims disagree with American values as well as policies, but that does not make mean they agree with bin Laden. In Secretary of Defence Donald Rumsfeld's calculus, the US cannot win if the number of people the extremists are recruiting is larger than the number we are killing and deterring, or convincing to choose moderation over extremism. To achieve this—to thwart our enemies, but also to reduce their numbers through deterrence, persuasion and attraction—required a better-balanced strategy.

In the information age, success is not merely the result of whose army wins, but also whose story wins. The struggle against extremist jihadi terrorism is not a clash of civilisations, but a civil war within Islam. We cannot win unless the Muslim mainstream wins. While we need hard power to battle the extremists, we need the soft power of attraction to win the hearts and minds of the majority. In this regard, presidential rhetoric about promoting democracy was less convincing than pictures of Abu Ghraib.

The Obama Administration came into office in 2009 committed to "smart power"—the combination of hard and soft power—or what Secretary of State Hillary Clinton called "using all the tools in our tool box". Soft power is the ability to obtain preferred outcomes through attraction rather than coercion or payments. I first developed the concept 20 years ago when I was trying to explain why I disagreed with the then widely accepted view that the US was in decline. After I estimated American economic and military resources, I realised that something was still missing: the ability to get the outcomes we wanted through attraction rather than coercion or payments.

During the Bush years, public opinion polls showed a serious decline in American soft power in Europe, Latin America, and most dramatically, across the entire Muslim world. Among the resources that produce soft power for a country are its culture (where it is attractive to others); its values, (where they are attractive and not undercut by inconsistent practices); and its policies (where they are seen as inclusive and legitimate in the eyes of others). When poll respondents were asked why they reported a decline in American soft power, they cited American policies more than American culture or values. Ironically, what the polls reported was just the opposite of Bush's claim that September 11 was a result of rejection of American values when in fact it was American policies that were at fault.

Reacting against the Bush strategy, the Obama Administration used declaratory policy and other adjustments to try to restore American soft power. But it also inherited a global financial crisis which originated in the US and contributed to a loss of both American hard and soft power.

As the US economy floundered and China continued to grow in the Great Recession of 2008-09, Chinese authors launched a flood of declinist commentary about the US, and they were not alone. In a 2009 Pew Research Centre poll, majorities or pluralities in 13 of 25 countries believed that China would replace the US as the world's leading superpower. Even the US government's National Intelligence Council projected that American dominance would be "much diminished" by 2025, and the one key area of continued American superiority—military power—would be less significant in the increasingly competitive world of the future. Russian President Dmitri Medvedev called the 2008 financial crisis a sign that America's global leadership was coming to an end.

How would we know if they are correct or not? To answer the question, we need to understand better what we mean when we speak of power, and how it is changing under the conditions of a burgeoning revolution in information technology and globalisation. We also need to avoid certain pitfalls.

First, one must beware of misleading metaphors of organic decline. Nations are not like humans with predictable life spans. Rome remained dominant for more than three centuries after the apogee of Roman power. Even then, Rome did not succumb to the rise of another state, but died a death of a thousand cuts inflicted by various barbarian tribes. Indeed, for all the fashionable predictions of China, India or Brazil surpassing the US in the next decades, the greater threats may come from modern barbarians and non-state actors. The classical transition of power among great states may be less of a problem than the rise of non-state actors. In an information-based world of cyber insecurity, power diffusion may be a greater threat than power transition.

States will remain the dominant actors on the world stage, but they will find the stage far more crowded and difficult to control. A much larger part of the population both within and among countries has access to the power that comes from information. Governments have always worried about the flow and control of information, and the current period is not the first to be strongly affected by dramatic changes in information technology. Revolutions are not new, nor is transnational contagion, or non-state actors. What is new—and what we see manifested in the Middle East today—is the speed of communication and the technological empowerment of a wider range of actors. This creates a much more complex setting for the exercise of American power in the 21st century. After all, non-state actors killed more Americans on September 11 than the government of Japan did with its attack on Pearl Harbour in 1941.

At an even more basic level, what will it mean to wield power in the global information age of the 21st century? What resources will produce power? In the 16th century, control of colonies and gold bullion gave Spain the edge; 17th-century Netherlands profited from trade and finance; 18th-century France gained from its larger population and armies; while 19th-century British power rested on its primacy in the Industrial Revolution and its navy.

Today, it is far from clear how we measure a balance of power in an information age. Many current projections of a shift in the global balance of power are based primarily on one factor—projections of growth in the gross national product of different countries. They ignore military and soft power, not to mention the difficulties of combining the different dimensions into successful strategies.

Take, for example, the recent events in the Middle East. The military retains important hard power, but alignment with the military alone can produce illusory short-term stability. If one seeks longer-term stability, a smart strategy also needs to have a soft-power narrative that attracts the information-empowered members of civil society in Tahrir Square.

Any net assessment of American power in the coming decades encounters a number of difficulties. Many earlier efforts have been wide of the mark, such as American estimates of Soviet power in the 1970s and of Japanese power in the 1980s. Today, some confidently predict the 21st century will see China replacing the US as the world's leading state, while others equally confidently argue that the US is only at the beginning of its power. But unforeseen events often confound such projections. There are a range of possible futures.

On American power relative to China, much will depend on the uncertainties of political change in China. Barring such uncertainties, China's size and high rate of economic growth will almost certainly increase its relative strength vis-a-vis the US. This will bring it closer to the US in power resources, but not necessarily mean that it will surpass the US as the most powerful country.

Even if China suffers no major domestic political setback, many current projections based on GDP growth alone are too one-dimensional and ignore US military and soft-power advantages, as well as China's geopolitical disadvantages in the Asian balance of power, compared with America's likely favourable relations with Europe, Japan, India and others.

My own estimate is that among the range of possible futures, the more likely are ones in which China gives the US a run for its money, but does not surpass it in overall power in the first half of this century. As Lee Kwan Yew once told me, China has the advantage of a talent pool of 1.3 billion people to draw upon, but the US can draw upon a talent pool of 7 billion and re-combine them in a diverse culture that is not limited by the ethnic Han nationalism.

If it remains open, America's culture of openness and innovation will keep it central in a world where networks supplement, if not fully replace, hierarchical power. The US is more likely to benefit from such networks and alliances.

On the question of absolute rather than relative American decline, the US faces serious problems in areas such as debt, secondary education, and political gridlock, but one should note that these are only part of the picture. It is important to look beyond current conventional wisdom. But while it is useful to question the short-term conventional wisdom of gloom and doom, it is equally important not to let preferences determine analysis. Among the negative futures are ones in which the US overreacts to terrorist attacks by closing inwards, thus cutting itself off from the strength it obtains from openness.

But barring such mistakes, in principle, and over a longer term, there are solutions to the major American problems that preoccupy us today such as long term debt (for example, consumption taxes that pay for entitlements); and political gridlock (for example, changes in redistricting procedures to reduce gerrymandering, changes in Senate rules, and so forth.) Of course, such solutions may forever remain out of reach. But it is worth distinguishing situations where there are no solutions from those which could in principle be solved.

Describing power transition in the 21st century as an issue of American decline is inaccurate and misleading. Such analysis can lead to dangerous policy implications if it encourages China to engage in adventurous policies or the US to overreact out of fear.

America is not in absolute decline, and it is likely to remain more powerful than any single state in the coming decades. At the same time, it will certainly be faced with a rise in the power resources of many others—both states; and non-state actors. It will also face an increasing number of issues in which solutions will require power with others as much as power over others.

American capacity to maintain alliances and create networks will be an important dimension of its hard and soft power, but a successful smart-power strategy will require better contextual intelligence in understanding the various dimensions of power than the US demonstrated in the first decade after September 11.