Beyond data access: The real AI challenge in Financial Services

The recent announcement that the London Stock Exchange Group (LSEG) will make its datasets available natively in Databricks through Delta Sharing has generated excitement across financial services. For good reason: market data has long been locked behind legacy delivery systems, and moving it directly into the Databricks environment removes significant friction.

But for banks, asset managers, and insurers, the challenge has never been just about data access. The real test is what happens after the data lands.

The pain points we hear from FSI leaders

When we speak with CIOs, CROs, and Heads of Data across banks, insurers, and asset managers, three persistent obstacles come up again and again:

1. Compliance and explainability: Regulations like Basel III, CCAR, MiFID II, and GDPR require more than accurate outputs. They demand transparent processes. Fully opaque black-box models face major obstacles in production deployment in regulated contexts, unless supplemented with explanation, auditing, or surrogate approaches. For many firms, demonstrating lineage across multiple systems remains a slow, manual process.
2. From POCs to production: Financial institutions don’t lack innovation. In fact, they often have dozens of AI pilots running, but the problem is scaling them. Governance, monitoring, and cost optimization are frequently afterthoughts, leaving promising projects stuck in “proof-of-concept purgatory.”
3. Fragmented data landscapes: Market data, client data, risk data, and operational data live in silos. Analysts often spend more time reconciling feeds than generating insight.

The three pillars of FSI-ready AI with Databricks

Access to high-quality, AI-ready data is necessary but not sufficient. To move beyond experimentation, FSI leaders need to build their AI strategies on three pillars:

1. A trusted data foundation: A single, governed source of truth where both internal and external datasets can be combined. This means clear lineage, fine-grained access controls, and auditability.
   Databricks’ Unity Catalog and Delta Sharing provide the backbone here, enabling transparent governance while reducing data integration friction. Note: lineage across disparate non-Databricks systems may still require external tooling or manual stitching.
2. Explainable, compliant models: AI in financial services must be auditable by design. That means embedding monitoring, bias detection, and explainability frameworks directly into the development lifecycle.
   Agent Bricks, while still relatively new, can help accelerate deployment in many cases, especially for standard tasks. Success, however, depends on pairing them with strong model governance and regulatory oversight.
3. Operational scale and resilience: Moving from POCs to production requires observability, cost control (FinOps), and reliability engineering. Without these, AI initiatives become fragile, expensive, and hard to defend to regulators or boards.
   The Databricks Lakehouse architecture reduces duplication of pipelines and supports resilient, scalable deployment when managed effectively.

From market data to trusted use cases

Seen through this lens, the LSEG–Databricks partnership is not just about faster access. It’s about enabling institutions to combine market data with internal datasets in a way that supports their most critical use cases:

• Risk management: Real-time liquidity stress tests and exposure monitoring.
  
• Trading analytics: Natural-language surfacing of portfolio insights that remain fully auditable.
  
• Compliance: Automated lineage can dramatically reduce reconciliation effort, cutting weeks of work to hours, though some domain-specific adjustments may still be required.

For decision makers, the real question isn’t “How quickly can I access LSEG data?” but “How do I ensure every AI-driven decision built on that data is trusted, compliant, and scalable?”

Actionable takeaways

1. Map regulations to data capabilities: Identify where your current systems fall short in terms of explainability and lineage. Prioritize closing those gaps before scaling new AI pilots.
2. Consolidate, don’t duplicate: Use open standards like Delta Sharing to reduce dependency on bespoke pipelines. Fewer moving parts mean faster audits and lower risk.
3. Embed governance early: Treat governance and monitoring as first-class citizens in AI development. Retrofitting compliance after deployment is costly and poses major risks.
4. Measure the POC-to-production gap: Track how many of your AI pilots graduate to production. If fewer than 10% succeed, governance, operations, and organizational confidence are often significant bottlenecks, although data quality, funding, or skills can also play a significant role.

Final words

The LSEG–Databricks story is an important milestone for the industry. But the institutions that will gain a real competitive advantage are not those who adopt market data faster. They are those who adopt it responsibly, with governance, compliance, and resilience built in from day one.

For financial services, the finish line is not data access. It’s achieving confidence in every AI-driven decision across risk, trading, compliance, and client engagement.

About DATAPAO

DATAPAO is a Data Engineering and Data Science consulting firm that supports the entire data journey, including strategy, implementation, training, and innovation.

As a trusted Databricks partner since 2016, DATAPAO is the one-stop solution to understand and leverage data better building on Databricks. DATAPAO tackles the most challenging data problems and helps organizations become truly data-driven.