Adapt or fall behind: a review of the Databricks executive survey on AI and its use
Understand how 600 tech leaders think about AI and find out how to future-proof your business before adopting the emerging tech.
Databricks Data + AI Summit 2025: Key Announcements
25 Jun 2025 · 30 min read
Budapest, June 24, 2025
The Databricks Data + AI Summit 2025 highlighted major advancements in the Databricks Data Intelligence Platform, reaffirming the company’s commitment to open standards, unified governance, and approachable AI.
Key Innovations and Enhancements:
- Lakebase (Public Preview): A fully managed, Postgres-compatible database for AI-native applications, deeply integrated with the Lakehouse.
- Databricks Apps: A serverless platform to build and scale interactive data and AI-powered applications.
- Enhanced AI Capabilities: Including MLflow 3.0, Agent Bricks (Beta), serverless GPU compute, and advanced vector search tools.
- Unity Catalog Improvements: Expanded support for Delta Lake and Apache Iceberg, advanced governance with ABAC, metrics, and lineage.
- Data Engineering Innovations: New offerings like Lakeflow Designer and Declarative Pipelines simplify and accelerate ETL development.
- User Experience Tools: AI/BI Genie reached general availability, and Lakebridge was introduced to streamline data warehouse migrations.
These advancements solidify Databricks’ leadership position in the evolving data and AI ecosystem.
Lakebase
Databricks Lakebase Public Preview
Summary of the feature:
Databricks Lakebase is now in Public Preview. It is a fully managed Postgres-based database, purpose-built for AI-native applications and tightly integrated with the Databricks Lakehouse. Lakebase introduces a modern OLTP architecture that separates compute and storage, offers serverless autoscaling (including scale-to-zero), and enables instant, Git-style branching. It leverages Neon, a recent Databricks acquisition, for copy-on-write branching and dynamic serverless compute.
Why is it important for customers?
Lakebase addresses key limitations of traditional databases, such as complex provisioning, poor scalability, and siloed analytical workflows. It simplifies infrastructure and accelerates development by enabling instant provisioning, elastic scaling, and seamless integration with the lakehouse. Customers can combine operational and analytical data to build intelligent applications, serving features and models, powering standalone applications, and analysing operational data. This allows organizations to combine operational and analytical data to power AI-driven applications. Lakebase is available on AWS and Azure (GCP support is planned).
It is optimized for data-intensive applications, particularly those built with Databricks Apps. However, it is not intended for general, mission-critical operational databases.
Typical workflow types:
- Data serving: Serve analytical data at a low latency to downstream applications.
- Application state storage: Manage workflow state using a transactional data store.
- Feature serving: Provide feature data with low latency for machine learning models.
Why is it important for Databricks developers?
Developers can leverage Postgres’s vibrant ecosystem, including extensions like PostGIS and pgvector. The separation of compute and storage allows for instant, zero-copy cloning of databases (branching), mirroring Git-like workflows for isolated test environments, which is economically friendly. Serverless autoscaling enables ephemeral database environments at virtually no cost or effort. It provides a unified development environment via the Databricks SQL Editor and integrates natively with Unity Catalog for simplified access control and identity.
You can select a Lakehouse table to sync automatically to a database instance. In this case, the database side is read-only.
Where you can read more:
Can be enabled directly from your Databricks Account.
- Can be enabled directly from your Databricks Account.
- Blog post: “Announcing Lakebase Public Preview“
- Blog post: “What Is a Lakebase?“
- Pricing information is available.
How did Neon work?
Neon architectures separates compute and storage. The Neon Control Plane manages cloud resources across both storage and compute.
A Neon compute runs Postgres, and storage is a multi-tenant key-value store for Postgres pages that is custom-built for the cloud.
Neon storage consists of three main components: Safekeepers, Pageservers, and cloud object storage.
Safekeepers are responsible for the durability of recent updates. Postgres streams Write-Ahead Log (WAL) to the Safekeepers, and the Safekeepers store the WAL durably until it has been processed by the Pageservers and uploaded to a cloud object store.
Pageservers are responsible for serving read requests. To do that, Pageservers process the incoming WAL stream into a custom storage format that makes all page versions easily accessible. Pageservers also upload data to cloud object storage and download the data on demand.
We can think of Safekeepers as an ultra-reliable write buffer that holds the latest data until it is processed and uploaded to cloud storage. Safekeepers implement the Paxos protocol for reliability. Pageservers also function as a read cache for cloud storage, providing fast random access to data pages.
Databricks Apps
Databricks Apps General Availability
Summary of the feature:
Databricks Apps is now generally available across 28 regions and all three major clouds, enabling customers to securely build, deploy, and scale interactive data and AI-powered applications natively on the Databricks Data Intelligence Platform. It provides a fully managed, serverless runtime with built-in identity, governance, and observability, supporting any popular Python or JavaScript framework (e.g., Node.js, React, Streamlit, Dash, Plotly, Gradio).
Why is it important for customers?
This feature solves the “last-mile challenge” of building data-heavy apps for analytics and AI by removing significant engineering effort typically required for deploying and managing apps (authentication, SSO, infrastructure, data access, observability, audit logs, environment management). Customers can rapidly deliver user-facing tools (LLM copilots, data quality dashboards, ops apps) directly within their governed data environment, without moving or duplicating data.
Why is it important for Databricks developers?
Developers can focus on building the application logic and the UI rather than managing the underlying infrastructure. They can utilise their preferred Python or JavaScript frameworks and seamlessly integrate with existing software development practices and tools like Git and CI/CD.
Where you can read more:
Anthropic’s Model Context Protocol (MCP) Support in Databricks
Summary of the feature:
Anthropic’s Model Context Protocol (MCP) is now directly integrated into the Databricks platform. MCP servers can be hosted with Databricks Apps, providing a seamless way to deploy and manage MCP-compliant services. Users can interact with and test MCP-enabled models directly in the Playground environment and connect their agents to leverage Databricks with hosted MCP servers for Unity Catalog functions, Genie, and Vector Search.
Why is it important for customers?
MCP support enables seamless deployment of standardised agent interfaces, allowing integration with AI models and enterprise tools without custom infrastructure. This simplifies secure access to enterprise data through consistent agent tooling.
Why is it important for Databricks developers?
Within a unified environment, developers can rapidly test MCP-enabled models, deploy custom MCP servers as apps, and integrate with Databricks-native features like Unity Catalog and Vector Search.
Details
Databricks’ Model Context Protocol (MCP) provides a standardised way for AI agents to connect with tools, resources, prompts, and contextual information. This standardisation allows a tool to be created once and used with any agent, regardless of its origin, and similarly, enables the use of external tools by your agents. Databricks integrates MCP directly into its platform, simplifying the deployment and management of MCP-compliant services without extra infrastructure, and permits testing MCP-enabled models in the Playground environment.
Databricks offers two main approaches for MCP servers:
1. Managed MCP Servers
These servers are pre-configured, hosted, and maintained by Databricks, designed to make agent connections to enterprise data easy. Unity Catalog permissions are automatically enforced, ensuring agents and users only access authorised tools and data.
Key managed MCP servers include:
- Vector Search: Enables agents to query Databricks Vector Search indexes within a specified Unity Catalog schema. For example, searching unstructured data in a `prod.customer_support` schema.
- Unity Catalog Functions: Allows agents to execute Unity Catalog functions within a designated schema. This includes custom Python or SQL data retrieval UDFs.
- Genie Space: Facilitates agents in querying a specified Genie space for insights from structured Unity Catalog tables.
2. Custom MCP Servers
Organisations can host their custom or third-party MCP servers as Databricks apps. This benefits from deploying and sharing existing MCP servers or running third-party solutions.
Where you can read more:
Mosaic AI
Agent Bricks (Beta)
Summary of the feature:
Agent Bricks is a new product in beta that allows enterprises to build high-quality, domain-specific AI agents by simply describing the task and connecting data sources. It automates the generation of evaluation suites and optimises agent quality and cost. Agent Bricks includes automatic evaluation (synthetic data, custom LLM judges), automatic optimisation (prompt engineering, model fine-tuning, reward models, Test-Adaptive Optimisation), and balancing cost and quality. Mosaic AI Research developed these optimisations, including Agent Learning from Human Feedback (ALHF), which translates natural language guidance into technical optimisations.
Why is it important for customers?
Agent Bricks addresses the complexity, evaluation difficulties, excessive tuning knobs, and cost/quality trade-offs often encountered in agent development. It delivers significantly higher quality and lower-cost agents than manual approaches, accelerating development timelines from weeks to days. It enables sustainable, scalable AI systems that provide consistent business value and allows domain experts to contribute to system improvement without deep technical AI expertise.
Why is it important for Databricks developers?
Developers can focus on defining the agent’s purpose and strategic guidance. It leverages MLflow 3 for automatic evaluation benchmarks and provides advanced algorithms like ALHF to translate feedback into system improvements intelligently.
Where you can read more:
Image: Databricks
MLflow 3.0 Release
Summary of the feature:
MLflow 3.0 unifies traditional ML, deep learning, and generative AI development on a single MLOps platform. It introduces GenAI capabilities, including production-scale tracing for over 20 GenAI libraries, revamped quality evaluation with LLM judges, feedback collection APIs and UI, and comprehensive version tracking for prompts and applications. It enables a complete GenAI development workflow: debug with tracing, measure quality with LLM judges, improve with expert feedback, track changes with versioning, and monitor in production.
Why is it important for customers?
MLflow 3.0 addresses fragmentation in tools and challenges in quality measurement and continuous improvement for GenAI applications. It provides high application quality, complete observability, and consistency across different types of AI initiatives. It empowers organisations to turn real-world usage and feedback into better evaluation datasets and refined applications.
Why is it important for Databricks developers?
Developers get consistent workflows, standardised governance, and production-grade reliability across all AI workloads. They can rapidly debug with traces linked to code, data, and prompts. MLflow’s LLM judges allow for systematic quality assessment, and the Review App enables structured expert feedback collection. Version Tracking and the Prompt Registry bring engineering rigour to managing application changes and prompt evolution. Deployment Jobs ensure that only validated applications meeting quality requirements reach production, integrated with Unity Catalog for governance.
Where you can read more:
Storage-Optimised Endpoints for Mosaic AI Vector Search (Public Preview)
Summary of the feature:
This new Vector Search engine is built for petabyte-scale data by decoupling storage from compute, leveraging Spark’s parallelism. It offers multi-billion vector capacity, up to 7x lower cost, 20x faster indexing, and SQL-style filtering. It’s a drop-in replacement for existing APIs and comes with full governance powered by Unity Catalog. It also includes direct integration in the Agent Playground for rapid RAG prototyping.
Why is it important for customers?
It enables customers to unlock more value from unstructured data for AI by building high-performance RAG, entity resolution, and semantic search systems across their entire data estate, without encountering hard capacity limits or ballooning costs. It makes running GenAI in production economically feasible across massive unstructured datasets.
Why is it important for Databricks developers?
Developers benefit from significantly faster indexing, enabling rapid iteration cycles. Using intuitive SQL-style syntax for metadata filtering simplifies operations, making migration easier due to consistent APIs. Full governance through Unity Catalog ensures proper access controls, audit trails, and lineage tracking. Developers can test and deploy chat agents backed by Vector Search in just two clicks through the Agent Playground.
Where you can read more:
Serverless GPU Compute (Beta)
Summary of the feature:
Databricks is introducing GPU support in its serverless platform, currently in beta with A10g GPUs and H100s coming soon. This fully managed service eliminates the complexity of GPU management, providing on-demand access without requiring long-term reservations. Users can run notebooks on serverless GPUs and submit them as jobs, all under the full governance of Unity Catalog.
Why is it important for customers?
GPU-powered AI workloads become significantly more accessible, allowing customers to focus on their AI models rather than operational overhead. Serverless GPU provides the necessary performance for training models, running inference, and processing large-scale data transformations.
Why is it important for Databricks developers?
Developers no longer need to manage complex GPU infrastructure, simplifying the deployment and scaling of GPU-intensive tasks. Full Unity Catalog governance applies to these GPU workloads.
Where you can read more:
AI Functions in SQL (Faster and Multi-Modal)
Summary of the feature:
AI Functions now allow users to access generative AI capabilities directly from within SQL, with dramatic performance improvements and expanded multi-modal capabilities. They are now up to 3x faster and 4x lower cost than other vendors on large-scale workloads. New functions like ai_parse_document enable effortless extraction of structured information from complex documents, working seamlessly across text, images, and other data types.
Why is it important for customers?
This enables customers to process large-scale data transformations quickly and efficiently, unlocking insights from previously hard-to-process enterprise content.
Why is it important for Databricks developers?
Developers can easily integrate generative AI capabilities into their SQL workflows, making complex data transformations more straightforward and more cost-effective directly within the data platform.
Where you can read more:
- Apply AI on data using Databricks AI Functions
- AI Functions in SQL: Bringing LLM and AI Power to Your Data
High-Scale Model Serving
Summary of the feature:
Databricks has enhanced its Model Serving infrastructure to support over 250,000 queries per second (QPS). A new proprietary in-house inference engine has been launched for LLM serving, up to 1.5x faster than properly configured open-source engines like vLLM-v1 on typical workloads.
Why is it important for customers?
This enhancement ensures production readiness for enterprise AI applications by supporting increased throughput and lower latencies for real-time online ML workloads. It makes serving LLMs on Databricks easier, faster, and often lower total cost than DIY serving solutions.
Why is it important for Databricks developers?
Developers can bring their demanding real-time online ML workloads to Databricks, with the platform handling infrastructure and reliability challenges, allowing them to focus on AI model development.
Where you can read more: We are waiting for the release of the Summit videos describing it.
Mosaic AI Gateway General Availability
Summary of the feature:
Mosaic AI Gateway is now generally available, serving as a unified entry point for all AI services. It provides centralised governance, usage logging, and control across an entire AI application portfolio. New capabilities include automatic fallback between providers, PII (Personally Identifiable Information) and safety guardrails, rate limit policies, and usage tracking.
Why is it important for customers?
Customers can implement robust rate limits, track usage, and enforce critical safety and PII guardrails on their AI workloads, regardless of whether they are running on Databricks or through external services. This ensures compliance and responsible AI deployment.
Why is it important for Databricks developers?
Developers benefit from a centralised management point for their AI services, simplifying control and monitoring across diverse AI applications.
Where you can read more:
Unity Catalog (UC)
Redpanda Iceberg Topics Integration with Unity Catalog
Summary of the feature:
Redpanda’s new Iceberg Topics now seamlessly integrate with Unity Catalog’s Managed Iceberg Tables and REST Catalog. This allows users to turn Kafka streams into entirely governed Unity Catalog-managed Iceberg tables in one step, delivering real-time lakehouse analytics without heavyweight connectors or custom ETL jobs. Iceberg Topics handle parquet batching, exactly-once commits, and Unity’s predictive optimisations.
Why is it important for customers?
This integration delivers real-time lakehouse analytics and slashes costs and operational effort by running streaming and high-throughput Apache Iceberg ingest on the same Redpanda cluster. It provides confidence in adopting an ‘Iceberg-forward’ data strategy without compromising the rich features of mature lakehouse products like Databricks. It eliminates the need for costly connector infrastructure or per-stream ETL jobs for high-value data in motion.
Why is it important for Databricks developers?
Developers get a single CRUD lifecycle and consistent user experience for topics-as-tables, managing data and schema as one entity using existing Kafka tools, the rpk CLI, Cloud REST APIs, K8s CRs, or Terraform provider. Unity Catalog’s built-in Schema Registry implicitly or explicitly defines the Iceberg table schema, and all streams are discoverable through Databricks tooling as both Iceberg and Delta Lake tables. Developers also get Kafka metadata (partitions, offsets, keys) as an Iceberg Struct for pipeline validation.
Where you can read more:
- Blog post: A Real-time Open Lakehouse with Redpanda and Databricks
- Redpanda Enterprise trial: https://cloud.redpanda.com/try-enterprise.
Unity Catalog Unifies Delta Lake and Apache Iceberg (Various Features)
Summary of the feature:
Unity Catalog is now the only unified catalog that works seamlessly across Delta Lake and Apache Iceberg formats, engines, and clouds. This includes full support for the Iceberg REST Catalog API (read GA, write Public Preview), Iceberg managed tables (Public Preview) with features like liquid clustering and predictive optimisation, Iceberg catalog federation (Public Preview) for governing/querying tables in AWS Glue, Hive Metastore, and Snowflake Horizon without copying data, and Delta Sharing for Iceberg (Private Preview) for sharing UC and Delta tables.
Why is it important for customers?
This addition eliminates the need to choose between Delta Lake and Apache Iceberg, breaking down artificial silos that restrict access to data and AI tools, fragment governance, and lock metadata. It delivers unmatched interoperability and eliminates format lock-in.
Why is it important for Databricks developers?
Developers gain unified governance across all data and AI assets. This simplifies the management and querying of data across different Iceberg sources and provides best-in-class performance for managed Iceberg tables.
Where you can read more:
- Blog post: What’s new with Databricks Unity Catalog at Data + AI Summit 2025
- Blog post: Announcing full Apache Iceberg™ support in Databricks
Unity Catalog Metrics (Public Preview)
Summary of the feature:
Unity Catalog Metrics, now in Public Preview on AWS, Azure, and GCP (GA later this summer), makes business metrics first-class assets in the lakehouse. Unlike BI-layer definitions, metrics defined at the data layer are reusable across all workloads, such as dashboards, AI models, and data engineering jobs, and are fully addressable via SQL.
Why is it important for customers?
This solves the long-standing problem of inconsistent metric definitions across tools and teams, often leading to confusion and a lack of trust in data. Certified metrics come with out-of-the-box auditing and lineage, enabling trusted and compliant insights across teams.
Why is it important for Databricks developers?
Developers can define metrics once in Unity Catalog and reuse them everywhere, across AI/BI Dashboards, Genie, Notebooks, SQL, and Lakeflow jobs, with upcoming integrations for BI and observability tools.
Where you can read more:
New Curated Discovery Experiences with Intelligent Insights (Discover, Domains, Certifications/Deprecation Tags, Request for Access)
Summary of the feature:
Unity Catalog is expanding its business-aware governance with a new Discover experience (Private Preview), a curated internal marketplace of certified data products organised by business domains (e.g., Sales, Marketing, Finance). This includes AI-powered recommendations, data steward curation, intelligent signals (data quality, usage patterns, relationships, certification status), and a built-in Databricks Assistant for natural language querying. Upcoming features include Domains for business-area organisation, Beta for Certifications and Deprecation Tags (via Tag Policies), and Public Preview for instant Request for Access to data assets.
Why is it important for customers?
This empowers business users by making trusted data easier to find, understand, and use. They can quickly assess the trust and relevance of data and streamline data delivery processes.
Why is it important for Databricks developers?
Developers can curate high-value data and AI assets, ensuring they are easily discoverable and understood by business users. Tag Policies help enforce consistent tag usage across the platform.
Where you can read more:
High-Leverage Governance with Scalable, Attribute-Driven Controls (ABAC, Tag Policies, Data Classification)
Summary of the feature:
Unity Catalog now provides intelligent automation and flexible, scalable controls for sensitive data governance. This includes Attribute-Based Access Control (ABAC) in Beta (AWS, Azure, GCP), allowing flexible access policies using tags at catalog, schema, or table levels. Tag policies (Beta on AWS, Azure, GCP) govern how tags are created, assigned, and used. Data classification (Beta on AWS, Azure, GCP) intelligently detects and tags sensitive data (e.g., PII) within 24 hours, automatically protecting it when used with ABAC.
Why is it important for customers?
This addresses the challenge of scaling data governance in growing organisations by automating controls and ensuring consistent policy enforcement. It reduces governance gaps, security risks, and operational bottlenecks. It minimises manual effort for PII detection and automatically protects sensitive data.
Why is it important for Databricks developers?
Developers can define policies using tags, simplifying the management of access controls. Automated data classification reduces manual PII detection efforts, and tag policies ensure consistency across the platform.
Where you can read more:
Automated Data Quality Monitoring at Scale
Summary of the feature:
Unity Catalog now intelligently detects and helps resolve data quality issues across all tables, in Beta on AWS, Azure, and GCP. It checks data freshness (how recently updated) and completeness (expected data volumes) using data intelligence across schemas.
Why is it important for customers?
Health indicators help customers understand the health of their data at a glance. Data owners can prioritise issues based on downstream lineage, uncover root causes, and set alerts using built-in logging and dashboards, ensuring high standards for data pipelines.
Why is it important for Databricks developers?
Developers benefit from automated monitoring, reducing the need for manual data quality checks.
Where you can read more:
Delta Sharing Innovations (Iceberg Interoperability, Network Gateway, MV/ST GA, ABAC, OIDC Token Federation)
Summary of the feature:
Delta Sharing now provides full Iceberg interoperability, allowing cross-platform sharing to any Iceberg-compatible engine. The new Delta Sharing Network Gateway simplifies network configuration for data providers. Streaming Tables and Materialised View (MV/ST) sharing is now Generally Available, enabling efficient creation and sharing of customer-facing data views. Attribute-Based Access Control (ABAC) for Delta Sharing is in Beta, allowing fine-grained access policies using tags. OIDC Token Federation is now Generally Available for enhanced security, enabling non-Databricks users to authenticate with custom Identity Providers.
Why is it important for customers?
These innovations expand secure, cross-platform data sharing regardless of cloud or platform, breaking down silos and driving innovation. They eliminate data duplication or file conversion, reduce network configuration complexity, and protect sensitive information with fine-grained access policies. Enhanced security and flexibility for authentication are also provided.
Why is it important for Databricks developers?
Developers benefit from seamless interoperability between Delta Lake and Apache Iceberg, simpler setup for external data access, and consistent governance across shared data.
Where you can read more:
Clean Rooms Enhancements (Identity Resolution, GCP GA, Multi-party Collaboration, Self-run Notebooks)
Summary of the feature:
Databricks Clean Rooms have been enhanced with privacy-centric identity resolution, enabling secure matching and enrichment of datasets without exposing raw PII. Clean Rooms are now Generally Available on Google Cloud Platform (GCP), facilitating seamless cross-cloud collaborations. Multi-party collaboration is now supported, allowing up to 10 organisations to work together in a single room across different clouds, regions, or data platforms. Clean Rooms also support secure self-runs, allowing collaborators to upload and run their notebooks with explicit approval from other participants.
Why is it important for customers?
Customers can securely match and enrich their data, combine operational data, and collaborate with partners without exposing sensitive raw data. The GA on GCP expands collaboration flexibility across major clouds, avoiding cloud silos and vendor lock-in. Multi-party support unlocks richer insights requiring more than two organisations. Self-run notebooks provide faster iteration, more autonomy, and full auditability within a trusted environment.
Why is it important for Databricks developers?
Developers can set up clean rooms that support complex, multi-party analytical scenarios while preserving privacy. Code governance features ensure no code runs without explicit consent, providing an audit trail and reducing risk.
Where you can read more:
- Blog post: What’s New with Data Sharing and Collaboration – Summer 2025
- Blog post: What’s New with Databricks Clean Rooms
Databricks Marketplace and Sharing Ecosystem Expansion (New Partners)
Summary of the feature:
Databricks is expanding its open ecosystem by welcoming new partners for data sharing solutions and advancing technology partnerships. New partners joining the Marketplace via Delta Sharing include SAP (coming soon), S&P Global Market Intelligence, FactSet, Integral Ad Science (IAS) (coming soon), Infor Nexus, Magnite (coming soon), Crunchbase, and SambaTV (coming soon).
Why is it important for customers?
Customers gain access to a broader range of enterprise data directly within Databricks, without needing third-party ETL tools or data replication. This includes financial intelligence, media quality datasets, supply chain network data, advertising activation, and predictive intelligence on private companies.
Why is it important for Databricks developers?
This expansion provides a broader array of high-quality, governed data assets that can be easily integrated into workflows, enhancing the utility and breadth of solutions built on Databricks.
Where you can read more:
Lakeflow
Lakeflow Designer (Private Preview)
Summary of the feature:
Lakeflow Designer is an AI-powered, no-code visual pipeline builder integrating with the Databricks Data Intelligence Platform. It supports drag-and-drop functionality and natural language prompts for creating ETL pipelines, and its outputs are Lakeflow Declarative Pipelines. The AI is grounded in the structure, semantics, and usage patterns of the user’s data (table definitions, column names, query history).
Why is it important for customers?
Lakeflow Designer enables business analysts to build scalable production pipelines and analyse data without writing code. It unifies business and data teams in a single environment, eliminating siloed workflows, redundant work, and coordination overhead. Pipelines are production-ready from day one with built-in observability, governance, and scale. The AI provides trustworthy, production-grade pipelines that align with existing workflows and governance standards.
Why is it important for Databricks developers?
Data engineers can review, understand, and improve pipelines generated by Designer without switching tools or rewriting logic because they are standard Lakeflow Declarative Pipelines (ANSI SQL). This results in smoother handoffs and fewer rebuilds.
Where you can read more:
- Blog post: Announcing Lakeflow Designer: No-Code ETL, Powered by the Databricks Intelligence Platform
Databricks Lakeflow General Availability
Summary of the feature:
Databricks Lakeflow, the unified data engineering solution, is now Generally Available. It includes three core components: Lakeflow Connect for reliable, managed ingestion (now with expanded connectors for enterprise apps, databases, file sources, data warehouses, and the new Zerobus API for high-throughput direct writes); Lakeflow Declarative Pipelines, built on the open Spark Declarative Pipelines standard with a new IDE for data engineering; and Lakeflow Jobs for native orchestration with advanced control flow and real-time triggers.
Why is it important for customers?
Lakeflow solves data engineering challenges posed by fragmented tool stacks by providing an end-to-end solution for ingestion, transformation, and orchestration on the Databricks Data Intelligence Platform. This eliminates complex tool integrations and extra licensing costs. It enhances data governance through deep integration with Unity Catalog, offering complete visibility and control over data pipelines. Zerobus enables simple, real-time event data ingestion directly into the lakehouse at very high throughput and near real-time latency.
Why is it important for Databricks developers?
Developers benefit from expanded built-in connectors, eliminating the overhead of custom connectors. Lakeflow Declarative Pipelines are 100% source-compatible with the open Spark standard and backwards-compatible with DLT, ensuring portability and smooth adoption. The new IDE for Data Engineering streamlines pipeline development with side-by-side code/DAG, instant data previews, context-aware debugging, Git integration, and AI-assisted authoring. Lakeflow Jobs offers comprehensive orchestration capabilities and end-to-end observability tools.
Where you can read more:
Donating Declarative Pipelines to Apache Spark (Open Source)
Summary of the feature:
Databricks is donating Declarative Pipelines to Apache Spark as a new open-source standard. This proven declarative API simplifies building robust data pipelines across batch and streaming workloads by allowing users to define “what” their pipeline should do and letting Apache Spark determine “how” to execute it. It includes declarative APIs for tables and transformations, native batch and streaming support, data-aware orchestration, automatic checkpointing, retries, and incremental processing, all with execution transparency.
Why is it important for customers?
This simplifies ETL and makes it more maintainable by automating “glue code,” reducing pattern reimplementation, and providing a standardised framework for testing, lineage, CI/CD, and monitoring. This significantly increases productivity, reliability, and maintainability for complex production pipelines.
Why is it important for Databricks developers?
Developers can write simpler SQL to describe desired results for full pipelines, as Spark handles optimal execution plans, dependencies, and incremental processing automatically. This makes pipeline development more straightforward, reliable, and collaborative, and serves as a common layer for the entire Apache Spark ecosystem to build upon.
Where you can read more:
AI/BI
AI/BI Genie General Availability
Summary of the feature:
AI/BI Genie is now generally available on all clouds. It allows business users to ask data questions in natural language and receive instant insights in text summaries, tabular data, and visualisations, explaining how answers were derived. Analysts can package data and semantics into Genie spaces for specific topics, acting as a local knowledge store. Genie Deep Research is also coming soon for complex, multi-step “why” questions, capable of producing research plans and analysing multiple hypotheses with precise citations.
Why is it important for customers?
Genie democratises data access for business users, empowering them to self-serve questions and get trustworthy, accurate, relevant responses that understand their organisation’s unique context. It handles large datasets without column count restrictions or complex instructions and integrates with Unity Catalog Metrics for accuracy. Users can also blend local spreadsheets with managed datasets for ad hoc analysis.
Why is it important for Databricks developers?
Data analysts can scale their impact by creating knowledgeable agents. Features like Benchmarks allow for systematic evaluation and improvement of Genie’s accuracy, and ‘Ask for Review’ fosters collaboration. Genie Conversation APIs enable embedding Genie into various collaboration tools or custom internal applications. With knowledge mining and extraction, the next-generation knowledge store simplifies curation and makes day-to-day usage a continuous learning pipeline.
Where you can read more:
- Blog post: AI/BI Genie is now Generally Available
Introducing Databricks One
Summary of the feature:
Databricks One is a new, visually reshaped experience designed to access data and AI with minimal friction for business users. It provides a single, intuitive entry point to interact with AI/BI Dashboards, AI/BI Genie, and custom Databricks Apps. Content can be browsed and organised by business domain, and it is built on Unity Catalog for trusted and secure access, ensuring data governance is maintained. The whole experience will enter public beta later this summer, with a “consumer access” entitlement available today.
Why is it important for customers?
Databricks One helps business teams make smarter decisions without needing to be technical experts. It streamlines access to relevant dashboards, apps, and Genie spaces through an intuitive interface and AI-powered recommendations. It ensures that data access is secure, compliant, and auditable from the start, leveraging existing Unity Catalog permissions.
Why is it important for Databricks developers?
Administrators can easily onboard business users with the new “consumer access” entitlement. It provides a unified platform for managing data access for technical and non-technical users, including row and column-level security.
Where you can read more:
- Blog post: Introducing Databricks One
Other
Databricks Asset Bundles now in the Workspace (Public Preview)
Summary of the feature:
Databricks Asset Bundles are now supported directly within the workspace UI (Public Preview), enabling teams to collaboratively build, version-control, and deploy projects from Git folders without needing the CLI. It supports modular development, Git integration, multi-environment deployments, and ensures that development jobs, pipelines, and resources automatically reference the latest uncommitted files for faster iteration.
Why is it important for customers?
This makes it easier for data scientists, analysts, and AI engineers to work interactively in the workspace with best practices like version control, testing, and CI/CD. It streamlines the development process, aligning work progression from development to production with standard software practices and making it accessible to a broader range of users.
Why is it important for Databricks developers?
Developers can now develop and deploy bundles entirely within Databricks, interacting directly through Git folders in the UI, eliminating the need for CLI or VS Code for many tasks. It provides a clear “Deploy” step for promoting changes and enables engineers, data scientists, and analysts to contribute to production workflows with built-in CI/CD readiness.
Where you can read more:
- Blog post: Announcing Databricks Asset Bundles now in the Workspace
- Example bundle repository: https://github.com/databricks/bundle-examples.
- Databricks Asset Bundles with UI in the workspace
Databricks SQL Performance Improvements (Predictive Query Execution and Photon Vectorized Shuffle)
Summary of the feature:
Databricks SQL (DBSQL) Serverless has achieved a 5x performance gain since 2022 and now adds 25% acceleration with the launch of Predictive Query Execution (PQE) and Photon Vectorized Shuffle. PQE introduces a continuous feedback loop within the query engine that monitors running tasks in real time and can replan stages on the spot to avoid wasted work and improve stability. Photon Vectorized Shuffle, a rewritten C++ engine shuffle, delivers 1.5x higher throughput in CPU-bound workloads like large joins by optimising memory access. These improvements roll out automatically at zero additional cost.
Why is it important for customers?
Customers benefit from significantly faster queries and dashboards (e.g., turning a 20-second dashboard into a 15-second one). PQE provides more predictable performance with fewer surprises, especially for complex pipelines and mixed workloads. All improvements are delivered automatically without any need for manual configuration or tuning.
Why is it important for Databricks developers?
Developers no longer need to spend time on manual performance tuning. We’ve seen the most significant gains on CPU-bound workloads, which can cut minutes off total runtime. DBSQL Serverless remains an easy-to-use platform with no infrastructure to manage, instant elasticity, and out-of-the-box optimisation.
Where you can read more:
Apache Spark 4.0 Release
Summary of the feature:
Apache Spark 4.0 marks a major evolution of the Spark analytics engine, bringing significant advancements across various areas. Key highlights include SQL language enhancements (scripting, session variables, reusable SQL UDFs, PIPE syntax, collations, parameter markers), significant improvements to Spark Connect (high feature parity, multi-language support including Go, Swift, Rust), reliability and productivity enhancements (ANSI SQL mode by default, new VARIANT data type for semi-structured JSON, structured logging), Python API advances (native Plotly-based plotting on PySpark DataFrames, Python Data Source API, polymorphic Python UDTFs), and Structured Streaming advances (Arbitrary Stateful Processing v2, State Data Source Reader, State Store Enhancements).
Why is it important for customers?
Spark 4.0 is more powerful, ANSI-compliant, and user-friendly, streamlining complex analytics workflows and ensuring stricter data integrity. The VARIANT data type offers schema flexibility and efficient querying of semi-structured data, simplifying ETL pipelines. Structured logging simplifies debugging and monitoring by integrating with observability tools.
Why is it important for Databricks developers?
Developers can leverage new SQL capabilities for more concise and maintainable code. Spark Connect improvements offer greater flexibility in language choice and a more modular architecture. Python users gain native plotting directly on PySpark DataFrames for easier data exploration, and can implement custom data sources and flexible User-Defined Table Functions (UDTFs) entirely in Python. Streaming applications benefit from more robust stateful logic, improved debuggability, and enhanced state store performance.
Where you can read more:
- Blog post: Introducing Apache Spark 4.0
- Related blog post: PySpark Native Plotting
- Official Spark 4.0 release notes, JIRA tickets, and community discussions for deeper deep dives
- Databricks Runtime 17.0 (which ships Spark 4.0)
Lakebridge (Free, Open Data Migration to Databricks SQL)
Summary of the feature:
Lakebridge is a new, free migration tool designed to automate and accelerate enterprise data warehouse (EDW) migrations to Databricks SQL. It automates up to 80% of migration tasks, covering profiling, assessment, SQL conversion (including stored procedures), validation, and reconciliation. It supports lift-and-shift and hybrid migration approaches and includes built-in dashboards for transparency. Future enhancements will include Mosaic AI-powered code conversion, a dedicated Data Migration module, and a graphical user interface (GUI).
The core technology is BladeBridge, which Databricks recently acquired. BladeBridge was the go-to migration automation tool with multiple supported source and target systems.
Why is it important for customers?
Lakebridge helps organisations modernise from legacy, siloed data warehouses to unlock faster insights, reduce costs, and consolidate analytics and AI workloads on Databricks’ open, unified platform. It addresses the complexity and risks of migrations, accelerating project timelines. It initially supports migration from over 10 legacy data warehouses and ETL tools, and converts proprietary syntax to open, ANSI-compliant SQL. It supports automated validation.
Databricks will undertake the development of BladeBridge as LakeBridge, but we still need to understand the exact roadmap. Exercise caution regarding out-of-the-box automation promises, as our experience with BladeBridge indicates that the degree of automation relies on numerous factors; consequently, it can vary significantly between codebases.
Why is it important for Databricks developers?
Developers can significantly reduce manual migration efforts, as Lakebridge automates many tasks. The tool is extensible, allowing Databricks partners to contribute features, and integrates battle-tested migration technology. It provides an end-to-end workflow from discovery to validation.
If you have used BladeBridge before, you know that you need to exercise caution. Understanding the analyser’s output is critical, as it can mislead your scoping.
Where you can read more:
- Blog post: Introducing Lakebridge: Free, Open Data Migration to Databricks SQL
- Partner Portal for partners
PySpark Native Plotting
Summary of the feature:
PySpark Native Plotting, available with Databricks Runtime 17.0, introduces built-in visualisation capabilities directly from PySpark DataFrames. It eliminates the need for users to convert PySpark DataFrames to Pandas API on Spark DataFrames for plotting. It supports various common chart types (line, bar, area, scatter, pie, box, histogram, density/KDE) and is powered by Plotly as the default backend, using native pandas internally. It leverages efficient data processing strategies like single-pass reservoir sampling and global metrics for accuracy and scalability.
Why is it important for customers?
This feature makes data exploration easier and more powerful, allowing customers to create intuitive plots directly from their PySpark DataFrames. It streamlines data analysis and makes it more accessible.
Why is it important for Databricks developers?
Developers can visualise distributions and trends from their datasets without leaving the Spark context or writing separate matplotlib/plotly code, significantly streamlining exploratory data analysis. The design aligns with the Pandas API on Apache Spark, providing a consistent and intuitive experience. Performance improvements include single-pass sampling and SQL-side optimisations for ML-based plots.
Where you can read more:
- Blog post: PySpark Native Plotting
- Mentioned in: Introducing Apache Spark 4.0
- PySpark API documentation for detailed guides and examples
Final Thoughts
The Databricks Data + AI Summit 2025 announcements demonstrate a significant stride towards unifying data and AI workflows, making them more accessible and powerful for both technical and business users. Innovations like Lakebase and Databricks Apps empower developers to build robust AI-native applications with ease, while advancements in Unity Catalog and MLflow 3.0 ensure comprehensive governance and streamlined MLOps for all AI initiatives.
Databricks is actively bridging the gap between technical and business teams. Features such as AI/BI Genie and Databricks One allow business users to interact with data and AI in natural language, gaining immediate, trustworthy insights. Similarly, Lakeflow Designer enables business analysts to build production-ready ETL pipelines without code, fostering collaboration and eliminating silos. The expansion of Unity Catalog’s discovery experiences with intelligent insights and domain-based organisation further empowers business users to find and utilise certified data assets effectively. This holistic approach ensures that organisations can leverage their data intelligence platform to drive value across all functions, from deep technical development to strategic business decision-making.
About DATAPAO
DATAPAO is a Data Engineering and Data Science consulting firm that supports the entire data journey, including strategy, implementation, training, and innovation.
As a trusted Databricks partner since 2016, DATAPAO is the one-stop solution to understand and leverage data better building on Databricks. DATAPAO tackles the most challenging data problems and helps organizations become truly data-driven.
, DATAPAO
