Apache Spark

Apache Spark Data Source V2 is Spark’s way to support native table formats, not only files. Szehon Ho walks through why that exists. Early Spark read and wrote file formats such as Parquet and used a thin Hive Metastore catalog. Apache Iceberg and Delta Lake added a much richer metadata layer so engines can run large, transactional operations. DSV2 expands that surface so a data source can plug that metadata into Spark, including through catalogs. 🎥 youtube.com/watch?v=MOQ3G8Ul… #ApacheSpark #OpenSource

We are happy to announce the availability of Apache Spark 4.1.4! 🎉 Spark 4.1.4 is the fourth maintenance release containing security and correctness fixes. This release is based on the branch-4.1 maintenance branch of Spark. We strongly recommend all 4.1 users to upgrade to this stable release. To download Spark 4.1.4, head over to the download page: spark.apache.org/downloads.h… Spark 4.1.4 is also available on PyPI (pyspark, pyspark-connect, pyspark-client) and the Maven Central Repository. To view the release notes: spark.apache.org/releases/sp… We would like to acknowledge all community members for contributing to this release. This release would not have been possible without you.

Agents are writing a ton of Spark code these days. Most of it is error-ridden, and the errors are the boring kind: a column that doesn’t exist, a table nothing upstream declares, a session config invented from thin air. At OSA Con 2026, @lisancao (@databricks) will cover declarative pipelines so structural mistakes get caught before data moves, Spark Connect so session context and credentials stay out of the agent’s hands, and skills and prompts that make one-shots more successful. She’ll also cover the tradeoffs between model choice, tokens, and deployment infrastructure. 🗓️ Monday, November 2 🕓 10:35-11:00 AM PST 📍 San Francisco + online · free 🔗 Session: osacon.io/sessions/2026/gett… #ApacheSpark #SparkConnect #OSACon

The core issue is that their source system is both relational and a document database. All of the documents are stored as XML. Figuring out how to handle that in production was a headache, because it is very challenging for end users to parse XML or JSON. That is where Variant comes in. Geir Alstad walks through this in a talk on Apache Spark Variant shredding in production. 🎥 Watch the full session: youtube.com/live/imEjS0UoZ2I #ApacheSpark

Agents don’t ask one question a day. They ask 10 or 100, against the same data stack and the same SLA. At Open Lakehouse + AI in Bellevue, Felix Cheung (@nvidia) walks through GPU-accelerated Apache Spark with no code change, plus @DeltaLakeOSS, @ApacheIceberg, and Spark Connect. 🎥 Watch: youtube.com/watch?v=OE59djsc… #OpenLakehouse #ApacheSpark

Apache Spark can analyze your query plans and derive the dependencies for you. You don’t need to declare them anymore. @lisancao sits down with Andreas Neumann on Spark Declarative Pipelines. Spark owns the graph, manages session lifecycle and writes, while @ApacheAirflow orchestrates Spark as an application. 🎥 Watch the full conversation: youtube.com/watch?v=dS5AwlC-… #ApacheSpark #ApacheAirflow #DataEngineering

In Apache Spark 4.2, retrieval (NEAREST BY), governed semantics (metric views), and agent access (Spark Connect) run inside the engine, next to the data and under the same permissions, instead of in separate services. Download Spark 4.2 👉 spark.apache.org/downloads.h… #ApacheSpark #AI #GenAI

Felix Cheung (Apache Spark PMC, @nvidia) and @lisancao on running Spark SQL and DataFrames on GPUs with NVIDIA’s cuDF plugin, no job rewrites. Snapchat’s A/B experimentation pipeline: 10 PB of new data a day, processed with Spark. On GPUs, 70-90% lower cost. Also Spark Connect, including how a Spark Connect Gateway can sit in front of multiple servers, GPU ones included. 🎥 Full talk: youtu.be/nL6NVivpIfQ #ApacheSpark #SparkConnect #cuDF

Spark SQL picks up several additions in Apache Spark 4.2, out now: SET PATH for object resolution, scripting cursors, QUALIFY for window results, SYSTEM.BUILTIN qualification, plus Theta sketches, time_bucket, and top-K max_by/min_by. More analytical work stays in SQL. 🔗 Download Spark 4.2: spark.apache.org/downloads.h… #ApacheSpark #SparkSQL #SQL

Apache Spark 4.2 is out. Python Data Sources let you build batch or streaming readers and writers in Python, no JVM or Scala, used through the standard spark.read interface. Spark 4.2 adds time and memory profiling so they're easier to tune and operate. 🔗 Download: spark.apache.org/downloads.h… #ApacheSpark #PySpark #Python

Auto CDC arrives in Spark Declarative Pipelines with Apache Spark 4.2: first-class SCD Type 1. Configure how change events update a target table, and Spark handles deletions and out-of-order events, in place of hand-written merge logic. 🔗 Download Spark 4.2: spark.apache.org/downloads.h… #ApacheSpark #CDC #DataPipelines #ETL

.@ApacheAirflow is a popular orchestrator, but it often has little visibility when a Spark job fails. At Airflow Summit 2026, @lisancao sat down with Andreas Neumann to discuss Spark Declarative Pipelines. Spark owns the pipeline graph, derives dependencies, and handles session lifecycle and writes, while Airflow orchestrates Spark as an application. 🎥 Watch the full conversation: youtube.com/watch?v=dS5AwlC-… #ApacheSpark #ApacheAirflow #DataEngineering

Headed to #CommunityOverCode Glasgow? 🏴󠁧󠁢󠁳󠁣󠁴󠁿 Oct 11–14 features Apache Spark sessions on: 🔸 Resource auto-sizing 🔸 Spark 4.2 heterogeneous executors 🔸 CDC patterns 🔸 PySpark transpilation 🔸 Agent-native observability 🔗 See the schedule: communityovercode.apache.org… #ApacheSpark

Grounding agents in trusted data is a theme in Apache Spark 4.2: NEAREST BY for retrieval, metric views for governed numbers, Spark Connect for access. Retrieval and metrics live next to the data, under the same permissions. 🔗 Download Spark 4.2: spark.apache.org/downloads.h… #ApacheSpark #AIAgents #RAG #GenAI

Schema-on-Read Without Regrets: Lessons Learned from Apache Spark Variant Shredding in Production nitter.gq/i/broadcasts/1nxnRBvAg…

Arrow-optimized Python UDFs are on by default in Apache Spark 4.2: existing UDFs take the faster columnar path with no rewrite, dropping the row-by-row JVM/Python serialization cost. Plus Pandas 3 and zero-copy handoff to Polars and @duckdb. 🔗 Download Spark 4.2: spark.apache.org/downloads.h… #ApacheSpark #PySpark #ApacheArrow #Python

Join us next Wednesday for Schema-on-Read Without Regrets! Geir Alstad (Gabler AS) and @newfront (@databricks) will walk through Spark Variant shredding in a real pipeline. 🗓️ Wed, Sept 30 · 8:00 AM PT 🎟️ Register: luma.com/open-z92a #ApacheSpark

Open Lakehouse + AI hits Bellevue in two weeks! 🎉 Felix Cheung (@nvidia) will cover GPU-accelerated Spark, Delta Lake, Iceberg, and Spark Connect. Micah Kornfield (@databricks) on a unified future for @DeltaLakeOSS and @ApacheIceberg, plus a panel on connecting the open lakehouse and AI ecosystem. 🗓️ Wed, Oct 7 🕜 5–9 PM PDT 📍 Databricks Bellevue Register: luma.com/open-qljr #ApacheSpark #ApacheIceberg

Apache Spark 4.2 is out, and it adds vector primitives to Spark SQL: distance and similarity functions, normalization, aggregation, and NEAREST BY (a top-K ranking join). Retrieval and recommendations run in SQL at Spark scale, next to your data. 🔗 Download Spark 4.2: spark.apache.org/downloads.h… #ApacheSpark #VectorSearch #SparkSQL