Banking on AI: Why Infra Decisions Will Make or Break FSI Innovation

The eager buzz around GenAI in financial services companies’ hallways has evolved into something more sober: a growing recognition that the infrastructure supporting AI initiatives may actually determine which institutions succeed and which fall behind.

At a recent Hong Kong CDOTrends roundtable, sponsored by Digital Realty in partnership with Telstra International, technology leaders from financial services firms examined the fundamental challenges of constructing resilient AI infrastructure capabilities while navigating a labyrinth of regulations and the insatiable hunger for power.

This wasn’t some abstract discussion. Held under Chatham House Rule, it examined the fundamental challenges reshaping how banks, insurers, and investment firms strategize their technology investments as AI becomes a prominent part of the industry landscape.

Coming to terms with AI infrastructure reality

While the spotlight often shines on the AI models themselves and their potential applications, the roundtable participants consistently circled back to three fundamental infrastructure anxieties: the geographical home of their data, the networking muscle required for seamless data movement, and the specialized hardware demands that legacy IT setups are ill-equipped to handle.

One digital bank executive noted, “For me, the main problem is the hardware. In existing data centers, we cannot physically support GPUs. The GPU server will take power from that entire rack. Even though we would like to purchase some GPU servers, we need to find new data centers that are even more powerful.”

As a result, Roddie Samuel, vice president of sales at Digital Realty, observed a seismic shift occurring in data center demands. “Data centers have been pretty much the same for the last 20 years. But now we’re seeing huge demands on the power densities, totally changing the environment,” he explained.

This surge in power density marks a dramatic departure from the norm. As Samuel pointed out, “It’s gone from some traditional banking workloads of maybe two kilowatts a cabinet to… people using over 100 kilowatts in a single cabinet. It’s not like an incremental change — it’s an off-the-charts change.”

In turn, the search for better data center facilities is pushing financial institutions away from managing their facilities — the modus operandi for traditional banks — due to strict data regulations.

“Every bank is getting out of operating their own data center,” Samuel observed. “Everybody used to run their own data centers in their own buildings, whether it’s in an office or on their own campus. That is completely gone. Every time they get the opportunity, they want to move it to commercial data centers, predominantly for [operational] risk reasons, but it also gives them much more flexibility.”

Navigating the cross-border labyrinth

For financial institutions with operations spanning borders, particularly between Hong Kong and mainland China, the infrastructure puzzle becomes exponentially more complex. One banking IT leader articulated their predicament: “When we talk about data cross-border, it’s possible for us to consolidate or send the data to the solution in China, or at the same time, whether we should adopt those solutions in Hong Kong — that’s a big question to us.”

Alice Ting, global head of banking, financial services, and insurance at Telstra International, agreed, highlighting that the financial services industry is focused on the cross-border dimension. “For connectivity requirements with China, we can work with our colleagues in Telstra PBS, which is the first part-foreign-owned joint venture granted a cross-provincial IPVPN license in mainland China, to provide insights and expertise on the local regulations and implementation support.”

AI simply increases this infrastructure complexity. Companies now grapple with where data should reside and how to efficiently move it across jurisdictions as they train their models to be more efficient and minimize hallucinations.

Political undercurrents further muddy these waters. A participant from a Chinese bank noted, “As a Chinese bank, we utilize Deepseek for our AI needs. We receive numerous solutions from our head office. However, all these centrally provided solutions must also align with local regulations.”

Infrastructure or use case? Why AI deployment failures occur

Recent studies suggest a staggering 78% of AI Proof of Concepts (POCs) fail to transition into full-scale production. This begs the question: are infrastructure limitations or poorly defined use cases the primary culprits?

Maybe we are using the wrong KPIs? One investment banking representative suggested a tiered approach to measuring AI success: “The easiest way to gauge the KPI of any AI initiative should invariably begin with efficiency. The subsequent measure should focus on business benefits. The most challenging metric to assess is customer experience.”

However, this executive also pointed to a different bottleneck in implementation: “The problem is likely the rapid pace of technological evolution. Last year, we invested in building and training our optical character recognition (OCR) model. Yet, in 2025, we’ve discovered that we no longer need to train it ourselves; readily available commercial solutions have surpassed our in-house efforts.”

This rapid technological churn is widening a talent gap. “There’s a significant disparity in the available talent in Hong Kong, especially when compared with mainland China,” the same participant highlighted.

Another senior insurance executive emphasized the often-overlooked aspect of governance in driving AI adoption. “Do you have a robust AI governance framework that integrates all the moving parts — data privacy, data governance, the AI teams, the infrastructure teams? Are these entities and the business collaborating under a unified framework to understand our objectives? Often, I find this crucial element lacking in organizations.”

Data gravity: The pull that shapes infrastructure strategies

One area where roundtable participants agreed on was the impact of data gravity, where large concentrations of data naturally attract more data and applications. It fundamentally shapes how financial institutions strategize their infrastructure placement.

“Data gravity is all about location; if you have a significant amount of data in a particular place, it will naturally draw in more data and applications, creating a stronger gravitational pull,” Samuel explained. “We leverage this concept to guide our data center location strategy. By observing global trends in data accumulation, we can strategically position our data centers in areas with the highest gravitational pull.”

The influence of data gravity has transformed the traditional concept of a single data center into sprawling campuses. “In Japan, our approach isn’t to build solitary data centers. Instead, we develop campuses capable of supporting 200 megawatts of capacity, rather than a single building. This trend is particularly pronounced as cloud providers seek to establish their presence within these same campus environments.”

How AI inference refactors infrastructure needs

The shift from the intensive computational demands of AI model training to the more streamlined process of inference — deploying these trained models for real-world applications — is bringing about a new set of infrastructure requirements for financial institutions.

“Inference is definitely less demanding on the infrastructure,” Samuel noted. “The bulk of the training happens in massive GPU clusters. With inference, the focus shifts to how you effectively utilize those models and integrate them with your data.”

Ting highlighted the evolving networking demands driven by inference: “From our perspective, the critical importance of the underlying network in ensuring the success of AI projects can be broken into two primary categories: the networking to support GPU clusters inside data centres, and the networking to support the transport of data between data centres. Latency sensitivity is becoming increasingly important on network routes between data centres, particularly in APAC, to help financial institutions transport large amounts of data to support their AI requirements.”

The partnership imperative

The sheer complexity of these interconnected challenges is driving a renewed emphasis on strategic alliances between infrastructure providers.

“From our perspective, data center requirements have undergone a dramatic transformation,” Samuel stated. “The connectivity aspect is now paramount. Organizations have distributed data estates, and this isn’t a trend that will suddenly reverse. Data is naturally distributed due to regulatory requirements and the need for proximity to applications. However, network technology has made significant strides with advancements like fabrics, and this is where our partnership with Telstra truly shines.”

Ting echoed this sentiment, outlining how financial institutions are adopting a more nuanced approach to infrastructure: “When we engage with customers on their digital transformation infrastructure, three enduring principles remain constant: the right fit, the right sizing, and the right location. The technology stack and the appropriate sizing are dictated by the specific workload. Furthermore, we’re observing a significant number of customers initiating their AI journey in the public cloud. However, at a certain point, economic considerations often favor a transition to their private or hybrid cloud infrastructure.”

As financial institutions continue to navigate the uncharted territories of their AI journeys, these intricate infrastructure considerations may ultimately prove to be the decisive factor, outweighing the choice of algorithms or the definition of initial use cases. The institutions that master the art of secure, compliant, and high-performance data management across geographical boundaries will likely emerge as the true AI leaders, irrespective of the specific models they ultimately deploy.

Curious to learn how Telstra International and Digital Realty can support your AI strategy in Hong Kong? Contact us at [email protected] today to explore how we can help drive your AI success and support your digital transformation.

Image credit: iStockphoto/pingingz