How to Register & Prepare If you're interested in advancing your skills with Databricks through a Specialist Session, here's a clear guide on how to register and what free courses you can take to prepare effectively. How to Begin Your Learning Path S...
- 1190 Views
- 0 replies
- 3 kudos
3 weeks ago
Hello all! The next BrickTalks session has been scheduled! In case you missed the last one, BrickTalks are live, expert-led sessions that give the Databricks Community members the opportunity to interact with our Bricksters, ask questions and get to ...
- 211 Views
- 1 replies
- 5 kudos
Monday
BrickCon 2025 | Dec 3–5, 2025 A community-driven conference built by and for Databricks users around the world. Get ready for an unforgettable experience at BrickCon 2025 — the first-ever community-led Databricks conference bringing together da...
- 944 Views
- 3 replies
- 4 kudos
2 weeks ago
Stay on top of customer feedback at scale! The digital world makes it easier than ever for customers to share their opinions, and each review can shape your business decisions. With Large Language Models (LLMs), you can quickly extract and summarize ...
- 124 Views
- 1 replies
- 2 kudos
Tuesday
We kept the October momentum going strong — with vibrant technical discussions, creative experiments, and deep dives into AI, streaming, and governance. Here’s a look at what stood out this week in the Databricks Community Fresh Content & Articles...
- 817 Views
- 1 replies
- 5 kudos
a week ago
As we close out September and step into October, it’s incredible to see how our top contributors continue to lead with consistency, clarity, and collaboration. Week after week, they’ve stayed at the forefront of discussions, setting the tone for know...
- 1554 Views
- 4 replies
- 9 kudos
3 weeks ago
Learn, Level Up, and Get Rewarded! Save the dates: October 10 – October 31, 2025! We're celebrating learning, growth, and career advancement with an exciting opportunity to level up in data engineering, data analysis, machine learning, and genera...
- 169950 Views
- 402 replies
- 117 kudos
08-07-2025
Community Activity
- 30 Views
- 1 replies
- 0 kudos
DLT Pipeline with unknown deleted source data
Hello.. I need help. So the context is : - ERP data for company in my group is stored in sql tables - Currently, once per day we copy the last 2 months of data (creation date) from each table into our datalake landing zone (we can however do full cop...
- 30 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Your solution #1 is very frustrating to me as well, for a number of reasons. Simply put, we have to be able to compare incoming data to target data for normal ETL operations. One way around this is to create a view of your target silver table, outsi...
- 0 kudos
- 81 Views
- 1 replies
- 0 kudos
Chennai Data Bricks Users?
We have using it any groups to notify upcoming events in Chennai? if yes please add me in that or suggest how to join in that group.If not why not someone to initiate to create a group and add who ever interested in data bricks meetup, so that will h...
- 81 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @rmohanrangan! We already have a Chennai user group that you can join. All information about upcoming events in Chennai will be shared in this group. To join, please open this link and click on the “Join Group Hub” button on the right-hand side...
- 0 kudos
- 16 Views
- 1 replies
- 0 kudos
Safe Update Strategy for Online Feature Store Without Endpoint Disruption
Hi Team,We are implementing Databricks Online Feature Store using Lakebase architecture and have run into some constraints during development:Requirements:Deploy an offline table as a synced online table and create a feature spec that queries from th...
- 16 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Additional Context:The feature spec created from the synced table is served through an endpoint, and we need to keep the same endpoint URL for consumers.After schema changes, we currently recreate the synced table and feature spec with the same names...
- 0 kudos
- 131 Views
- 5 replies
- 11 kudos
(Episode 3: Hands-on API Project) - Learning Databricks one brick at a time, using the Free Edition
Episode 3: APIsLearning Databricks one brick at a time, using the Free Edition.Project IntroWelcome to everyone reading. My name’s Ben, a.k.a BS_THE_ANALYST, and I’m going to share my experiences as I learn the world of Databricks. My objective is to...
- 131 Views
- 5 replies
- 11 kudos
Latest Reply
- 11 kudos
This is great, thanks for sharing Ben, will share with my data community.
- 11 kudos
- 14 Views
- 1 replies
- 0 kudos
Unable to Add Multiple Columns in Single ALTER TABLE Statement on Iceberg Table via Unity REST Catal
Hello Databricks Team,I have implemented code to integrate the Iceberg Unity REST Catalog with the Teradata OTF engine and successfully performed read and write operations, following the documentation at https://docs.databricks.com/aws/en/external-ac...
- 14 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
This error stems from the Iceberg table metadata update constraints enforced by the Unity Catalog's REST API. Specifically, the Iceberg REST Catalog currently does not support multiple schema changes in a single commit. Each ALTER TABLE operation tha...
- 0 kudos
- 3085 Views
- 3 replies
- 1 kudos
How to identify which columns we need to consider for liquid clustering from a table of 200+ columns
In Databricks, when working with a table that has a large number of columns (e.g., 200), it can be challenging to determine which columns are most important for liquid clustering.Objective: The goal is to determine which columns to select based on th...
- 3085 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
@Alberto_Umana is it possible to get from system table the columns used in joins & filters of a table being queried?
- 1 kudos
- 1247 Views
- 2 replies
- 0 kudos
Multiple schedules in workflow with different parameters
I have a notebook that takes a file from the landing, processes it and saves a delta table.This notebook contains a parameter (time_prm) that allows you to do this option for the different versions of files that arrive every day.Specifically, for eac...
- 1247 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
You can do multiple schedules with Cron expression. If you are using a Cron expression in Databricks asset bundle YAML, but the limitation is you can't have one running at 0 past the hour and another at 25 past.i.e: quartz_cron_expression: 0 45 9,23 ...
- 0 kudos
- 9 Views
- 0 replies
- 0 kudos
Looking for Databricks–Kinaxis Integration or Accelerator Information
Hi Databricks Community,I’m looking for information on the partnership between Databricks and Kinaxis. Specifically:Are there any official integrations or joint solutions available between the two platforms?Does Databricks provide any accelerators, r...
- 9 Views
- 0 replies
- 0 kudos
- 1603 Views
- 1 replies
- 1 kudos
Delta table size not shrinking after Vacuum
Hi team.Everyday once we overwrite the last X month data in tables. So it generate a every day a larger amount of history. We don't use time travel so we don't need it.What we done:SET spark.databricks.delta.retentionDurationCheck.enabled = false ALT...
- 1603 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi, any update on this? Thanks
- 1 kudos
- 84 Views
- 2 replies
- 1 kudos
Migrating Talend ETL Jobs to Databricks – Best Practices & Challenges
Hi All,I’m currently working on a Proof of Concept (POC) to migrate existing Talend ETL jobs to Databricks. The goal is to leverage Databricks for data processing and orchestration while moving away from Talend.I’d appreciate insights on the followin...
- 84 Views
- 2 replies
- 1 kudos
- 2359 Views
- 1 replies
- 0 kudos
Batch reading from sql server tables with cdc on ssql server tables
Hi all,I need to do a batch load from sql server into Databricks. I have CC enabled on some tables. The simple appears to be union CDC and regular table to get a single set of records to load, but this appears to be fraught with risk of out of sequen...
- 2359 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
For Q1, Consider this approach or a watermark based approach:https://learn.microsoft.com/en-us/azure/databricks/ldp/cdcFor Q2,You have few options: Pushdown via JDBC Query, Use a SQL View instead if you want to avoid replicating the logic in spark.
- 0 kudos
- 14 Views
- 1 replies
- 0 kudos
Databricks DLT Joins: Streaming table join with Delta table is reading 2 Billion records per batch
Databricks DLT Joins: Streaming table join with Delta table is reading 2 Billion records from Delta Table for each and every Micro batch.How to overcome this issue to not to read 2 Billion records for every micro batch.Your suggestions and feedback w...
- 14 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@gudurusreddy99 Root CauseWhen performing stream-static joins, which means joining a streaming source with a static Delta table, Spark will scan the entire Delta table for each micro-batch if the static table is not filtered or incrementally joined...
- 0 kudos
- 7748 Views
- 5 replies
- 1 kudos
Spatial Queries
Hi,I'm trying to execute the following code:%sqlSELECT LSOA21CD, ST_X(ST_GeomFromWKB(Geom_Varbinary)) AS STX, ST_Y(ST_GeomFromWKB(Geom_Varbinary)) AS STYFROM ordnance_survey_lsoas_december_2021_population_weighted_centroidsWHERE LSOA21CD ...
- 7748 Views
- 5 replies
- 1 kudos
Latest Reply
- 1 kudos
@Corar You might want to enable that explicitly by setting 'spark.databricks.geo.st.enabled' configuration to value 'true'.
- 1 kudos
- 63 Views
- 0 replies
- 1 kudos
Databricks Clean Rooms: Bringing Oil & Gas Legacy Code to Modern Data Collaboration
The oil and gas industry invests over $600 billion annually in upstream activities, much of it supporting complex, proprietary models and simulation algorithms. Many organizations still depend on legacy C++ and C libraries for critical functions such...
- 63 Views
- 0 replies
- 1 kudos
- 18 Views
- 0 replies
- 0 kudos
Access to Databricks Volumes via Databricks Connect not working anymore
Hi all, I use the extension to debug my python code regularly and since yesterday accessing files in the Databricks Volume isn't working anymore. The situation in the UI of Databricks is as follows:When I execute a glob statement to list all zip-file...
- 18 Views
- 0 replies
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2113 | |
| 1154 | |
| 947 | |
| 935 | |
| 768 |
Latest from our Blog
Databricks Clean Rooms: Bringing Oil & Gas Legacy Code to Modern Data Collaboration
The oil and gas industry invests over $600 billion annually in upstream activities, much of it supporting complex, proprietary models and simulation algorithms. Many organizations still depend on lega...
Databricks Assistant Data Science Agent: Autonomous Analytics for Petroleum Engineers
Petroleum engineers spend hours wrangling and analyzing data to estimate well production targets. We demonstrate how the Data Science Agent can significantly accelerate this type of analysis and many ...
Azure Databricks — Serverless Private Connectivity to Customer’s Resources (Part-1)
Authors: Kiran Anand, Suraj Karuvel Introduction The serverless model provides customers with a fast and easy option to run their workloads in Azure Databricks. In this model, the compute runs in Dat...
How to Deploy Agent-to-Agent (A2A) Protocol on Databricks Apps?-> End-to-End Guide
Modern AI systems rarely live alone. They collaborate. Google’s Agent-to-Agent (A2A) protocol gives agents a common language (JSON-RPC over HTTP/S, with streaming via SSE) plus a discovery mechanism (...
