Setting direction around AI – vision, design and plan

Summary

Until recently, there was no single voice or source of leadership for AI within government. The Central Digital and Data Office, Government Digital Service, i.AI, and Department for Science, Innovation and Technology each held a different mandate around AI and were spread across different parts of Whitehall.

Due to this fragmented leadership structure and the absence of political sign-off for publishing the CDDO’s AI vision, digital leaders told us that there was limited strategic alignment around AI. There was simply not a coherent, well-understood, and strategic vision for AI. This meant government also performed poorly on the design and plan lenses – there remains too much vagueness and uncertainty thus far.

The other major concern we identified is that Whitehall has focused too much on regulation and risk management around AI, with relatively little emphasis on adopting and embedding new tools to improve public services. Important as AI safety is, a balanced approach is needed. The fact that there are only 74 deployed AI use cases (among the 87 government bodies who responded to a National Audit Office survey this year) shows how much further there is to go.

Vision

When attempting to transform and modernise government, the first step is developing a clear and realistic vision that’s well understood. Around AI, we found a mixed picture – with interviewees’ self-reported scores for this at a two or three.

Those across government broadly recognise the potential AI offers. There’s growing interest among senior civil servants and politicians – regardless of their political leanings. Indeed, as one digital leader memorably told us: “The level of ambition and energy from ministers is like nothing I’ve ever seen before in tech… They’ve been told, these are literal magic beans, and you’ve just got to sow the magic beans, and all your problems go away.”

In reality, all this excitement isn’t matched by a well-defined vision yet. There’s limited articulation, both inside and outside government, of what AI can and should mean as a tool for transforming and modernising public services.

This appeared to stem from two problems. The most fundamental was a perception that the model of AI leadership in central government that existed until July 2024 in Whitehall was fragmented and confusing.

There were four different agencies operating to different mandates – DSIT leading on ‘policy for AI’, GDS charged with scaling up digital tools across government, the Cabinet Office’s Central Digital and Data Office (CDDO) as the lead on ‘AI for policy’, and i.AI as the incubator for new innovations. This was a highly complex picture.

Some defended this as a sensible division of labour and, for those working on AI, it probably was clear and comprehensible. But many digital leaders accepted that it was too complex for those outside. With fragmented leadership, it was even harder for a coherent vision to break through.

This is exacerbated by the fact that departments are pushing ahead with exploring AI for their own purposes. This was partly a reflection of that lack of central leadership. As one digital leader explained: “I don’t want to sit around waiting for CDDO to tell me what to do, or not do anything until they come up with the perfect vision, because I sort of think that might never quite happen.”

As departments push on, they are developing their own visions for AI. This might entail anything from a fully fledged vision to a vaguer set of ideas about AI. And these visions are informed by their own department’s digital leaders and their preconceptions. In fact, some parts of government are not planning to develop visions at all. The National Audit Office found that 21% of government bodies have an AI strategy and 61% plan to develop one, but 15% have no intention of doing so at all.

One digital leader suggested that there are “no massive frictions” between these different departmental visions. This is positive. But they added that the lack of coherence still creates complexity: “[It’s] deeply confusing having so many different voices not quite saying the same thing.”

In other words, there is no coherent, shared vision for AI in government as things stand – a finding that the National Audit Office also made earlier this year. Its conclusion is hard to ignore: “There are risks to value for money if the government does not establish which department has overall ownership and accountability for delivery of the strategy for AI adoption in the public sector…”

A single voice in government was therefore needed to establish a well-understood AI vision. And so, the government’s announcement – just a few days after coming into office – that DSIT will now host all of CDDO, GDS and i.AI is promising. It establishes the kind of single AI authority that’s needed to build a whole-of-government vision.

The second problem we identified around vision is much simpler. It turns out that the CDDO has developed a vision for AI in government: one that some within the system, especially in digital roles, already know about. However, the CDDO has not received political sign-off to publish it (at the time of writing).

New-in-post Labour ministers will understandably wish to provide feedback on this document and possibly make changes. But DSIT should ensure this revised vision is published urgently. Delivering and promoting a clear vision should be the flagship statement of the new AI hub, resetting the government’s approach in this area.

Design and plan

With limited work around the fundamentals of vision-setting, it’s unsurprising that progress on design and planning has been mixed too.

Digital leaders working in the centre do have some clear ideas of exactly how and where AI should be being deployed. One suggested AI should be used in four areas: improving the user experience for government services; reducing cost; supporting workers with co-pilot capabilities; and improving use of data in decisions. Another articulated a similar matrix of sensible applications for AI organised around two variables: proximity to citizens and complexity
(see page opposite).

But across government, fully formulated plans for implementing AI are in short supply. One interviewee told us: “I don’t think anybody’s yet got what we would class as a proper AI adoption plan.” Another suggested that different types of AI-related activity aren’t being clearly delineated or understood. AI tools for improving operational delivery are conflated with externally facing AI
developments, such as overall national AI policy and regulatory approaches.

Without a central vision, no system-wide adoption plan is on the cards. Even though there are some AI tools already in use or late-stage development – we trialled Redbox (see appendix) and were impressed – implementation is generally limited. The NAO found that 37% of government agencies deploy AI already, but this was usually only with one or two use cases. In total, they found just 74 deployed AI use cases across 87 government bodies who responded to their survey – compared to over 7,000 digital services listed on the GOV.UK website. And interviewees suggested that most AI tools used in government today have been developed by external
bodies, rather than within the system.

It’s true that government has gone much further around AI safety and thinking about risk. But this mismatch is a problem. One digital leader told us: “All our policy work is going into stopping
AI and all our delivery work is going into enabling AI.” This is the analogy we described earlier – “It’s like we’re driving a car with one foot very firmly pressed down on the accelerator and one foot pressed down on the brake.”

It’s not that AI safety work is itself a problem. The issue is that, at the moment, the foot on the brake is pressed down much harder and much more consistently than the foot on the accelerator ever is.

This may be because risk management is of higher concern in the public sector and so comes naturally to Whitehall. For government services, it’s essential that AI tools are very accurate. The public expect their interactions with the state to be handled to a high standard and that information provided by their government is trustworthy. This can create understandable blockers to rapid rollout of AI. 

For example, the deployment of a generative AI chatbot on the GOV.UK website has been hindered by “answers [which] did not reach the highest level of accuracy demanded” for government – according to a blog published by GDS data scientists and user researchers.

The same blog post noted that: “Some users underestimated or dismissed the inaccuracy risks with GOV.UK Chat, because of the credibility and duty of care associated with the GOV.UK brand.” This illustrates the significant challenges involved in deploying some types of AI.

However, more generally, there can be too much emphasis on risk management and AI safety. One interviewee argued that “civil servants can be rather obsessed with risk”. They added: “I do wonder whether the fact the AI safety agenda has been so well-promoted really just speaks to the fact that it’s something civil servants were ready to grasp onto: the risks of AI we should be protecting against.”

For now, policy for AI is undoubtedly ahead of AI for policy (or operations, for that matter). With little evidence of a coherent vision – let alone a design or plan – the vehicle has remained stationary.

Departmental drivers – collaboration and accountability

Summary

Successful cross-government working has long been a challenge within Whitehall, reflecting the institutional design and incentives that exist in the system. With powerful departmental ‘fiefdoms’, an underpowered centre, and accountability resting ultimately with departmental ministers and permanent secretaries, it’s rarely easy to build a whole-of-government approach to cross-cutting issues.

With AI, we found a similar story. There are some networks and spaces in which both formal and informal collaboration take place and these are productive structures to build from. But they ultimately seem to be about discussion and engagement primarily, rather than genuine cross-departmental working.

The creation of the new AI hub within DSIT – putting the CDDO, GDS and i.AI all in the same place – offers a single source of leadership on this agenda. There may be some scope for it to drive a more collective approach to AI within Whitehall.

Interviewees were clear that one area where this is especially important is data sharing. By connecting data up across government, AI tools can reach their full potential and draw on a much richer range of material about citizens. The new AI hub should seek to drive collaboration in this area as a key priority.

Collaboration

Building a cross-government approach to AI requires effective collaboration, especially between departments and the centre. In Whitehall, there’s a tendency to believe that the UK is world-leading in this regard, but the reality doesn’t always bear this out. To borrow a term used by one Caribbean digital leader in a GGF project examining the transformation challenges governments face there, we often engage in “collaboration-by-mouth”. The rhetoric around working together is greater than the evidence of genuine collaboration.

The barriers to collaboration in the UK government are well-known; they are inherent in our system of government. One digital leader described Whitehall as defined by “loosely coupled fiefdoms, all motivated by public service and innovation”. They added: “The incentive structure of government is still such that collaboration is your lowest priority. That’s just fundamentally true.” Overcoming this isn’t straightforward.

We did find some spaces where collaboration – or at least information-sharing – is taking place. Interviewees told us about a cross-government forum for AI policy and a general AI forum where the CDDO shares the work it is doing with departments. There’s a digital, data and technology (DDaT) board attended by permanent secretaries and an AI practitioner community convened by CDDO (with representatives from outside government). 

We also heard about chief technology officer (CTO) meetings of various sizes.

The largest is a ‘council’ attended by all CTOs across Whitehall. There’s then a smaller version for CTOs from large operational departments and the CDDO, followed by a final meeting of the same members – but without the CDDO included.

There are some informal networks too. One digital leader described themselves as being part of an “accidentally arising community”, having connected with other public servants around a shared interest in AI. Having both these formal and informal spaces to collaborate and discuss AI can only be a good thing – especially given how early government is on its AI journey.

But there is a risk again of this ‘collaboration-by-mouth’, with much talking but little action. One interviewee made exactly this point – the high-level alignment is easier than the harder collaboration on practical issues: “I think we’re good at aligning on vision, but when you look at design and other [lenses], there are institutional barriers and hurdles to working effectively together.”

Accountability

This is where we encounter questions of accountability. Whitehall is defined by those departmental fiefdoms, with the centre often struggling to be influential. Civil servants are ultimately answerable to their departmental ministers. Permanent secretaries are the accounting officers – meaning they are accountable to Parliament for the work of their department.

This creates obvious incentives with predictable consequences. One digital leader asked the provocative question: “How do you break a system that, for hundreds of years, has been designed around departments doing things that they’re individual accountable leads for and instead [get them to] deliver genuinely cross-organisational stuff?”

The truth is that it’s difficult, but not impossible. In government, I saw how effective the centre can be in driving transformation around digital. We showed departments the value of coming on the digital journey – using carrots and sticks – and built a reasonable coalition of support to help
deliver it. Systems changed accordingly, after much hard work.

There are similar examples of this type of approach in Whitehall today. The government’s 2022-25 Roadmap for Digital and Data includes six formalised cross-Whitehall missions. One of these, ‘sponsored’ by the permanent secretary of HMRC, is that: “All departments will confirm an adoption strategy and roadmap for One Login by April 2023 and their services will have begun onboarding by 2025.” It’s an example of straightforward milestones matched with clear accountabilities – with responsibilities to take action across the system.

The new AI authority in DSIT is still taking shape, but it should try to build on this mission’s framework as set out by the CDDO. Its placement in a department which has genuine heft and extremely close links into the broader AI community is a good thing, but it’s not yet obvious how it will work to drive government-wide transformation. Learning from that existing CDDO roadmap is crucial as it begins to puzzle out its emerging role.

One area where the hub could offer tremendous value is around government data: specifically, making it far more interoperable. A recent article by Sir Robert Chote, chair of the UK Statistics Authority, raised exactly this topic. And data sharing is a crucial priority in the context of AI too, as one interviewee told us: “If government wants to be serious about using machine learning tools and frameworks to help it become more productive and efficient, it has to solve its data problems. Just limping along is no longer an option.”

Whitehall’s “unremediated legacy” around data, to use a term from one interview, is a serious challenge. Training and using AI models relies on the supply of high-quality data, both to create and keep improving these powerful tools. And many of the most exciting applications of AI can only reach their full potential if they can pull data from different parts of government. A personalised AI assistant for citizens is of limited use if it’s able to use Home Office data, but can’t access anything from the Department for Work and Pensions. The perennial Whitehall challenge around data sharing is one of the greatest threats to government’s AI transformation.

Perversely, this presents an opportunity: data’s key role in supercharging AI might focus minds and incentivise more collaboration. But the real risk is that things don’t change and poor data practices remain a significant barrier. To return to the accelerator and the brake metaphor, a failure to fix government’s data is like operating with a speed limiter firmly attached to the vehicle. We can only go so fast without ameliorating this challenge.

This is why the new AI hub in DSIT must focus on this agenda. One interviewee called for legislation to help this, telling us: “The first thing you [should] do is put a big digital bill through Parliament that removes all the impediments to data sharing, all the impediments to building user experiences that are entirely based on departmental silos.”

Various legislative efforts have already been explored in recent years. The Digital Economy Act 2017 took some steps to improve public services through better use of government data. And the new government has announced plans for a Digital Information and Smart Data Bill in the most recent King’s Speech, which included a commitment to reform data sharing and standards – again for the benefit of public services. It’s too early to tell whether this will finally realise
that crucial ambition.

What is clear is that a Big Bang is needed; in fact, it’s long overdue. Legislation may help, but the
government’s AI hub should also play a key role here. Departments must be held accountable for failing to share data and those common standards must be established to help make collaboration easier – it cannot be ‘by-mouth’.

Developing an AI-ready workforce – transformation leadership and people

Summary

It’s well-understood that workforce – already a challenge in the context of digital – is even more difficult to get right around AI. Getting the right skills in place within government, given the pay premiums that the private sector can offer, isn’t easy at all.

Despite that, Whitehall has some excellent AI talent (especially within i.AI), according to interviewees. But demand far outstrips supply and is certain to keep doing so. There’s little evidence that enough is being done to develop the AI skills needed within Whitehall, whether via technical training or upskilling around prompt engineering.

The absence of AI-specific roles within the Government Digital and Data Profession Capability Framework, though apparently being resolved, is another significant oversight. And more work needs to be done to understand when to rely on external skills and when to build tools internally.

When it comes to leadership, top civil servants outside of digital roles – such as permanent secretaries – must have a good understanding of AI. Interviewees noted the key role those leaders play in helping ministers understand what’s really achievable around AI in their departments. They must be well-informed so that they can balance hype with practicality.

For now, top officials have a variable understanding of AI, with some upskilling efforts in place that appear to be productive. Boosting understanding of this transformative technology among those leaders (and senior officials just outside that top tier) is another area that Whitehall needs to focus on.

Transformation leadership

For now, top officials have a variable understanding of AI, with some upskilling efforts in place that appear to be productive. Boosting understanding of this transformative technology among those leaders (and senior officials just outside that top tier) is another area that Whitehall needs to focus on.

One interviewee argued that the “number one issue is senior leaders’ lack of digital fluency”. These “generalist digital leaders” (as they put it) are “de-facto digital leaders because they make all the decisions around major change programmes and where money goes. But they would be horrified to be called ‘digital leaders’ because they think it’s something that’s done over on the periphery, on the edges, and it’s not important.”

In other words, a good understanding of what’s possible around AI, data and digital cannot solely be the preserve of those in strictly digital roles. It’s a prerequisite for everyone in Whitehall – and one that should be viewed as central, not peripheral – precisely because AI and data touch everything that government does today.

And having this level of understanding also helps officials better serve ministers. There was a perception from interviewees that politicians are only hearing the most optimistic voices
around AI. Senior officials must be able to give a sense of what’s actually achievable when advising ministers – as one interviewee explained: “Some permanent secretaries might be the
people with the first shot at framing those conversations. They need to have a level of understanding of the tech and what’s realistically possible, so we don’t get signed up to a vision
that we can’t deliver.”

Permanent secretaries are not a homogenous group in terms of their skills, backgrounds or capabilities. Around AI, it’s the same story. One interviewee described them as a “mixed knowledge group”, with some well-informed and others less on top of this agenda.

We heard about several existing efforts to upskill top officials. There was an away day held for permanent secretaries at Oxford University, involving a series of AI workshops. Other examples included a top civil servant visiting Cambridge University to learn more about AI, knowledge
exchange meetings between senior officials and leaders at companies like Google and Microsoft, and teach-ins where directors general and permanent secretaries heard from internal digital experts.

These are all worthy endeavours, but the overarching perception was that they aren’t moving the dial enough. One interviewee suggested that top leaders do “know AI’s a big bet” but “there’s no materiality” beyond that. Another suggested that Whitehall leaders are around five years behind
private sector CEOs: “There’s a big digital gap and then [there’s] an even bigger AI understanding gap.”

These challenges aren’t confined to the very top level, however. Understanding and enthusiasm
around AI varies among leaders beyond permanent secretaries.

One senior leader admitted: “If you picked up the phone to a random member of middle leadership [in this department] – SCS1s/deputy directors – some of them would be able to articulate a chunk of what I just said [about AI and digital], but quite a lot of them wouldn’t.” Another interviewee offered a similar account, describing a mix of progressive and unconvinced DGs: “There are people who are threatened by it, there are people who are cynical about it, and there are people who want it all – but want to do it for themselves because they don’t want to lose independence.”

Getting leaders on the same page around AI is essential. This might mean some remain sceptical of its transformative potential. That’s to be expected. But ensuring that the techno-optimists and the AI sceptics have a strong base of knowledge to work from is the key priority.

People

The last of the lenses presents an even greater challenge than transformation leadership. ‘People’ refers to a wider set of the workforce and their skills, knowledge and capabilities – getting this element right is especially difficult with AI.

Given the global race for AI talent and the public-private competition for these skills, it’s unsurprising that Whitehall struggles to keep up. Pay in the public sector is never going to match what’s available elsewhere and the pull factors – around public service and interesting work – can only go so far.

Despite that, we heard that i.AI had successfully attracted many talented people. The chance to join an environment already stocked with impressive talent, greater pay flexibility, and proximity to the PM/No.10 were all offered as drivers of this. Whether i.AI can continue to recruit as effectively now that it’s within DSIT (i.e. further from No.10) is yet to be seen. Beyond i.AI, one interviewee told us there are “pockets of exceptional talent”, including strong capabilities in some specific departments. 

But overall, government does not have the digital skills it needs, including around AI. One minister’s revelation in February 2024 that 20% of DDaT roles in his department were unfilled highlights this problem. More widely across Whitehall, interviewees pointed to three factors contributing to the AI capability challenge. 

First, the Government Digital and Data Profession Capability Framework – which defines DDaT roles and the skills that underpin them – has not been updated to reflect AI jobs. Departments use the existing framework to recruit some AI talent, often by adapting analogous DDaT job specs, but there are no AI-specific roles included in that document, nor any competencies mapped out for them.

We were told that new guidance to incorporate AI jobs is being developed, but it’s not clear when that will be in place. And as one interviewee pointed out, defining exactly what ‘AI skills’ are isn’t easy, with the relevant career and training pathways remaining somewhat obscure. Hiring the wrong kind of expensive AI talent creates challenges of its own, so getting a revised framework right is essential.

Secondly, several interviewees referred to effective ‘hiring freezes’ within departments which prevent them buying in new talent – including in DDaT roles. This follows the previous government’s plan to ‘cap’ civil service headcount, which has led to a number of external recruitment drives being frozen.

One senior leader from a Whitehall department told us: “It’s ridiculous to have that headcount number prevent us from being able to do more things in-house at better value, just because it
theoretically increases the total number of civil servants.” The new government scrapped this headcount cap in July 2024, which should enable greater flexibility to recruit AI (and digital) talent when it’s needed.

Finally, training to help upskill civil servants around using or developing AI isn’t good enough. One interviewee called this available training offer essentially “unusable”.

There are different AI skills that different cohorts of civil servants will benefit from. Some need technical training so that they can learn to develop or refine AI tools themselves.

Others, like policy professionals, need competence in things like prompt engineering, so they can be intelligent users of AI applications created by others.

When I was in government, GDS trained over 10,000 staff as part of our digital transformation goals. There’s little evidence of the same kind of ambition around AI. We agree with the perspective of one interviewee: “There’s got to be a huge upskilling programme.”

Having a good capability base will allow Whitehall to use or access AI much more effectively. In some cases, the intelligent client model will be best – relying on trusted private partners who can offer expertise that cannot be maintained internally. In other cases, internal development of AI tools can save money and prove more effective. But Whitehall can only follow either of these paths if it boosts its people’s capabilities around AI and develops a much-improved training offer.