Harnessing the future of supercomputing: The impact of the ExCALIBUR Programme

A transformative programme  

In an era where cutting-edge simulations and high-performance computing (HPC) are pivotal to solving global challenges, the ExCALIBUR programme has played a transformative role. Funded by the Strategic Priorities Fund (SPF) the £45.7 million programme was led by the Met Office and the Engineering and Physical Sciences Research Council (EPSRC), from October 2019 to March 2025. ExCALIBUR set out to (re)design next-generation simulation algorithms, codes, and tools to maximize the potential of future supercomputers. 

The need for exascale computing 

Prior to the start of the programme, supercomputing was on the brink of a revolutionary leap, transitioning into the realm of exascale computing. As highlighted by the Met Office Chief Scientist, Professor Stephen Belcher in September 2018, existing simulation codes – used in critical areas like weather forecasting, climate projections, and engineering – were ill-suited for the upcoming wave of exascale machines. These new computing systems required not only innovative software techniques, but also a highly skilled workforce of software engineers and researchers who could harness their full potential. ExCALIBUR was established to address this urgent need, ensuring that the UK remained a leader in simulation-based research. 

The programme's goals and objectives 

ExCALIBUR’s primary objectives were as follows: 

1. Redesigning simulation codes: to optimise them for exascale computing environments. 

2. Enhancing scientific productivity: by improving the speed, resolution, and complexity of simulations. 

3. Training a new generation of Research Software Engineers (RSEs): ensuring the workforce is ready to manage these new computing systems. 

The programme aimed to tackle high-priority scientific and engineering problems, and ensure that the UK’s computational infrastructure could handle vast data sets and complex simulations, especially with the advent of exascale supercomputers. 

Achievements and progress 

ExCALIBUR has made significant strides across various research fields, from meteorology to biomedicine, physics, and AI. The programme funded 110 projects, engaging over 250 scientists from 114 organizations globally. Key outputs have included advancements in simulation codes, algorithms, and models, as well as the development of new software tools and resources. 

The programme has also contributed to over 164 peer-reviewed publications and 264 dissemination events, enhancing the visibility of the UK scientific community and attracting new collaborations. A notable achievement is the development of 45 new code releases, which have already demonstrated performance improvements for existing simulation models, contributing to better use of current and next-generation supercomputers. 

The role of Research Software Engineers (RSEs) 

One of ExCALIBUR’s standout features has been its focus on strengthening the UK’s pool of skilled RSEs. These engineers are crucial in the development and deployment of HPC applications. ExCALIBUR has provided 100 RSEs with opportunities to collaborate with scientists and experts, enhancing their skills and career development. However, the programme also revealed challenges, particularly in retaining RSEs long-term, given the prevalence of fixed-term contracts. 

Addressing the UK’s supercomputing needs 

At the time the ExCALIBUR programme was conceived, the UK had 20 supercomputers in the Top 500 global rankings, contributing 2% of global HPC capacity. With the development of new supercomputers like Dawn and Isambard-AI, and the UK’s commitment to increasing its computational power by 20 times by 2030, ExCALIBUR’s innovations have laid the groundwork for the future. 

The programme has also proven essential for future-proofing the UK’s HPC capabilities, ensuring that new supercomputers can handle exabytes of data and complex simulations. Without ExCALIBUR, the UK’s scientific simulation community would struggle to make the most of these upcoming advancements, either relying on outdated systems or underutilizing next-gen machines. 

Evaluation and long-term impact 

Evaluations of ExCALIBUR, which began in 2022, highlight its success in meeting its goals. The programme has not only improved scientific productivity, but also fostered interdisciplinary collaborations and boosted the skills of RSEs. Notably, one-third of partnerships formed during ExCALIBUR have led to new collaborative projects. 

Though the full impact of ExCALIBUR will take years to materialise, the immediate benefits are clear. The programme has provided essential knowledge, tools, and networking opportunities, helping position the UK as a global leader in HPC and simulation software. The ExCALIBUR projects have already made a lasting impact on the fields of climate science, engineering, AI, and biomedicine. 

Key lessons from ExCALIBUR include the importance of public funding for complex scientific research and the need for a systematic approach to knowledge transfer and career development for RSEs. Future initiatives should continue building on ExCALIBUR’s success, ensuring the UK remains at the forefront of HPC and simulation capabilities. 

Conclusion 

ExCALIBUR has significantly advanced the UK’s ability to leverage supercomputing technologies, positioning the nation for leadership in exascale computing. By focusing on simulation software, cross-disciplinary collaboration, and RSE development, the programme has laid the foundation for the next generation of scientific research. However, continued investment and strategic planning are crucial to sustaining this momentum, and realizing the full potential of the UK’s HPC capabilities in the years to come.