This was always going to be one of the more eagerly awaited presidential memoirs in living memory. Not only because of Barack Obama’s astonishing success in his rise to the White House in 2008 as the first African-American president, and not just because of his proven eloquence in the spoken and written word.

The expectation derives as much from a quest for understanding as to how, after the country elected him twice – a figure self-described as being from "everywhere and nowhere" – the United States then turned to Donald Trump.

And Obama provides the answer, even if it is the spectre haunting this historical feast. Looking back to early 2007 and the beginning of his presidential campaign on that cold day in Springfield, Illinois, in the land of Lincoln, he cannot help but allow nostalgia to take hold. He recalls the feeling then that "we would catch lightning in a bottle and tap into something essential and true about America".

A Promised Land book cover

It becomes a powerful metaphor for his presidency. Obama knows, even if he finds it painful to admit, that the uplift he gave the country was "fleeting". His memoir becomes a study in the gradual recognition that the politics he believed in, founded on a common humanity, would struggle to heal the country’s deep divisions. That his story about what Americans "might be" was only ever a short-term break from an unrelenting cynicism towards politics and institutions that had already tapped deep roots in the country. He recognises, too, the fear and anger in what he calls the "meat and potato folks", many of whom "didn’t trust a word" he said. Still, Obama wanted to believe the "ability to connect was still there". The resilience of his optimism rarely falters.

With its reflective tone, its searing examination of his own failures and shortcomings, its account of his idealism meeting the blunt reality of a dysfunctional Washington intent on bringing him down, the book can be read as a kind of elegy for the "common thread" he sees binding society together. Obama is watching it unravel even as he is trying desperately to repair the selvedges in the American fabric.

Obama knows the history of "how this post war consensus broke down", from LBJ’s loss of the South following the Civil Rights Act, through Vietnam, feminism, Roe v Wade, busing, gay rights, Newt Gingrich and the Clinton impeachment.

Obama was fond of the quote that the "arc of the moral universe is long but it bends towards justice". Yet his inheritance – a global financial crisis and two intractable wars in Iraq and Afghanistan – was always going to mean that the arc could only be bent so far.

He recognises that Sarah Palin’s nomination as Vice-President provides a "template for future politicians", calls the rise of the Tea Party a "new and suddenly potent force", but concedes the working and middle-class whites gravitating towards it "had suffered for decades from sluggish wages, rising costs and the loss of the steady blue-collar work that provided secure retirements".

They might have been straight from Billy Joel’s 1982 song Allentown: "killing time, filling out forms, standing in line" in the places where the "restlessness was handed down". Obama stoked those embers of fear and anger himself when he once remarked that people in the small towns of Pennsylvania "cling to guns or religion or antipathy towards people that aren’t like them".

Obama was fond of the quote that the "arc of the moral universe is long but it bends towards justice". Yet his inheritance – a global financial crisis and two intractable wars in Iraq and Afghanistan – was always going to mean that the arc could only be bent so far.

If America was becoming even more riven at home, played out here in his recounting of toxic debates over how to staunch the fiscal bleed from the Wall Street Crash, healthcare, the fate of the auto industry along with tax and immigration reform, America’s image overseas was a different story. The country was "still a symbol abroad", he writes, still receiving the "rituals of tribute to an empire".

It is telling that the sheer madness of the Trump-led "birther" conspiracy reaches its peak as Obama deliberates on whether to send in the Navy SEALS to eliminate Osama bin Laden.

Obama’s rise had gained new momentum when in 2002, during his term as an Illinois State Senator, he had spoken out against the Iraq war in a speech of moral courage and simple wisdom. He was not opposed to all wars, he said, but, against the weight of political and public opinion at the time, including most in his own party, he had had been brave enough to worry aloud about a US occupation of "undetermined length, with undetermined costs and with undetermined consequences".

Discussing his toughening stance on the US/China trade imbalance, Obama draws on a phrase he’d surely redraft in a new edition, saying he felt like he was "haggling over the price of chickens at a market stall rather than negotiating trade policy between the world’s two largest economies".

In office, then, he wanted to temper visions of American manifest destiny with a "humility about our ability to remake the world in our image". He chafed at what he calls the Washington "playbook" where, clearly forgetting Vietnam, military chiefs prioritised "resolve" over the dangers of "mission creep" in Afghanistan. The antiwar candidate ended up putting more troops into the field than he brought home, but made the generals sign on the dotted line to put an end to "Pentagon freelancing" for the remainder of his presidency.

Consumed by the Middle East, Obama believed that the China challenge was still "decades away", that any threat to American hegemony would come, too, from Washington’s own "strategic mistakes". And, discussing his toughening stance on the US/China trade imbalance, he draws on a phrase he’d surely redraft in a new edition, saying he felt like he was "haggling over the price of chickens at a market stall rather than negotiating trade policy between the world’s two largest economies".

If there had been a subtitle for Obama’s memoir it would surely have been "an intellectual in politics". He confesses a tendency towards too much navel-gazing, admits to getting "lost in his head". He knows that at times the best that leaders and policymakers can do is to adapt to the world at large piecemeal, by groping with a genuine indetermination in their aims. There is in this work a weighing of options, a balancing of the alternatives, a measured commitment of the self.

For Obama, history both illuminates and irritates the present. He wanted to break free of the simple linear narrative, to throw down the gauntlet to the view that history had a "predetermined course, an endless cycle of fear, hunger and conflict, dominance and weakness". But he also recognised his limitations. He knew that what Czech President Vaclav Havel told him in Prague, that he had been "cursed with people’s high expectations", was probably right.

This book, which covers only Obama’s first term, carries the reader deep into the policy brief, up onto the podium and the "sugar high" of his speeches and into the life of a man trying to be a husband and father at the same time as running the country.

In the book’s preface Obama sets out the stakes at play. Writing amidst the global pandemic, and as President Trump scorched a very different path across the country, Obama will not give up on an interconnected world. We will learn to cooperate again, he muses, come to respect each other’s dignity, or "we will perish".

Obama’s presidency has for the past four years been assessed largely in the shadows of what followed it. But his memoir will begin the important process of assessing it on its own terms.

A Promised Land
Barack Obama
Viking