Like all good historical detective stories this one began with a handful of clues in a public archive and from chance encounters with people who were once eyewitnesses to secret events.

It was a warm spring day in 2015 when I made my way into London to share a platform with a former senior Intelligence Officer at the headquarters of an international think tank. I was there to give an academic overview of the Northern Ireland Troubles; he was there to offer a practitioner’s-eye view of how the British state neutralised the threat posed by the Provisional IRA.

The IRA was one of the most ruthless terrorist organisations in the final decades of the Twentieth Century, emerging from civil disturbances in Belfast and Derry in 1969-70 to conduct a highly sophisticated terror campaign against the local Unionist authorities. London intervened, proroguing the Stormont parliament and introducing direct rule combined with a huge uplift in British troops and intelligence capabilities. After a 35-year campaign the IRA finally called a halt to its ‘armed struggle’ in 2005.

Officer JB, I shall call him, explained to a largely foreign audience how intelligence operations turned the tide on the IRA’s terror network and brought its violence to an end. This narrative had a familiar ring to it. I had heard it before, a few years earlier, in interviews and focus groups I conducted with former police officers from the Royal Ulster Constabulary (RUC). What I hadn’t heard before was just how vital a role strategy, leadership and teamwork – three significant dimensions of much of my professional teaching and research – played in the success of Britain’s secret intelligence war against the IRA.

I was intrigued and wanted to find out more.

By a quirk of circumstances, I found myself again speaking about the UK’s experience of Counter Terrorism, this time at a high level conference in Egypt. At an undisclosed location in Alexandria, I gave an overview of the British Army’s 38-year-long Operation Banner, the military deployment in support of the RUC.

On a hot and humid September morning, I addressed a cavernous lecture theatre crammed full of hundreds of military officers from a range of nations who were attending a major multinational exercise. I talked to them about the concept of the Northern Ireland ‘Intelligence Pyramid’, ironic given the location, and how this Counter Terrorism model had been built by the British in their coordinating attack on the IRA. There is nothing secret about this – scores of books, articles and declassified documents have detailed the intelligence machinery utilised by the Security Forces in Northern Ireland.

What had not been explored in any great detail was why that system had been built and how it worked.

The questions I had directed at me by delegates from the US, Egypt, France, Chad, Uganda and many more countries convinced me that it was time to write a more fulsome account of covert action and intelligence operations during the Troubles.

If I was talking to overseas audiences about my research, then why shouldn’t I share that with people closer to home?

A few weeks later I found myself on another overseas assignment, this time at the Nigerian Army’s Resource Centre in Abuja, Nigeria, where I gave a high-level briefing to the country’s senior military and intelligence officers. I looked at three case-studies, the British in Northern Ireland, the United States in Iraq and the Russians in Syria. My argument was that force is only one means by which states can respond to the challenges posed by Violent Non-State Actors. For Counter Terrorism to really succeed, it must coordinate all of the tools of Grand Strategy for the purpose of accomplishing the ends of government policy.

Much the same as in London and Egypt, I argued that we have a responsibility to understand not only our opposition but also ourselves if we are to learn lessons from strategic history. As Sun Tzu’s famous aphorism runs, “if you know neither the enemy nor yourself, you will succumb in every battle.”

I still recall a senior delegate at the Abuja conference, a Russian Colonel, telling me he enjoyed my paper, intriguingly just before I stood up to deliver it. The mere fact that the distribution of the paper had allegedly been tightly controlled prior to the event did not seem to spare me the blushes of having argued in it that Russian military theorists were all too aware of how they often had to conduct deep operations from a position of geo-political weakness.

This brief exchange convinced me that the UK’s adversaries might have more of an interest in the lessons from Britain’s past, even if the British are not always interested in knowing these details themselves.

It occurred to me after all three occasions that, even though I was operating as an independent academic availing only of open sources, it was incredibly important to remain on top of my brief. My qualifications have always been a hard-earned streetwise intuition mixed with a decent university education and 20 years of academic research experience. This has enabled me to undertake advanced empirical-based investigations where other, more conventionally minded scholars, fear to tread. It is also tinged with a conviction that the Professional Military Education environment is a ‘safe space’ where I can exercise my own academic freedom and professional judgement.

Understandably, in order to remain at the graduate level of thinking and analysis, I have to spend a considerable amount of time in libraries, in the archives and in interviews with eyewitnesses to the historical episodes I write about.

With no special privileges or access, I have had to join the long queue of researchers lodging Freedom of Information (FOI) requests with the British government in the hope that they might declassify once-secret documents on Northern Ireland. Since returning from overseas trips to Egypt and Nigeria, as well as many other places, I have had to send off scores of FOI submissions. Some have been successful; the vast majority have not been.

Much of the material declassified and released to me in recent years has been truly eye-opening.

That the British state is protective of its secrets comes as no surprise to those of us who have been working with government archives for a quarter of a century. I have spent many years examining official documents on Northern Ireland and on its overseas colonies, and I know how incredibly opaque the state has been about its activities in places as far-flung as Malaya, Cyprus and Aden/South Yemen. As a result, I have had to rely on innovative Open Source Intelligence (OSINT) techniques to build up an accurate picture of the ‘ground truth’ of security and intelligence operations. I should also say, however, that I use these techniques when considering the point of view of Violent Non State Actors, including terrorist, insurgent and organised crime groups. Indeed, while they have not deposited a comparable paper trail, what they have left behind for researchers are other traces from the past.

One of the greatest assets that open source researchers have at their disposal, therefore, is people.

Disappointingly some of the people I contacted in veteran Security Forces circles were initially downbeat about my project. A few were reluctant to talk to me thanks to the chill factor the Provisional Republican Movement helped to manufacture by way of the Boston College Tapes Affair. The road ahead seemed long and lonely. As ever I relied on a small band of amazingly talented friends, acquaintances and fixers for assistance. It is this network who have been indispensable to my research. With their help, I managed to track down former agents and handlers, as well as well-wishers who believe the Security Forces have been denied a fair hearing amidst a deluge of sensational allegations about their conduct. As Lord Peter Hennessy, the doyen of studies on the British secret state, once remarked, ‘There is more fantasy per square inch about the British intelligence world than anything except the British Royal Family’.

By early 2018, I had signed with my publisher, Merrion Press, to write a new book, Agents of Influence. I was determined that the book would reflect the real experiences of those who held the line against terrorism. For far too long, some of my sources told me, the convention of ‘Neither Confirm, Nor Deny’ has enabled a range of conspiracy theories to flourish unchecked. Agents of Influence is a genuine attempt to put historical evidence into the public arena so as to inform the debate over the steps an ancient liberal democracy has taken in order to meet the threat posed by a toxic strain of violent extremism.

Above all, it is an attempt to evaluate the successes and failures of the UK in countering the pernicious threat posed by extremism and terrorism during the long years of the Northern Ireland Troubles.