Pinned Tweet
I've pretty much been over at Bluesky, catch you there ;)
bsky.app/profile/lisancao.bsโฆ
Lisa N. Cao retweeted
Sometimes the most important step is simply ๐๐ต๐ผ๐๐ถ๐ป๐ด ๐๐ฝ.
Women+ in OSS Day on Monday was a great reminder of that ๐ Open source can feel intimidating from the outside, but this event made one thing clear: every contribution ๐บ๐ฎ๐๐๐ฒ๐ฟ๐. Code, documentation, testing, reviews, questions, or simply helping someone get started all count.
Beyond the talks, people rolled up their sleeves and contributed to OSS projects like #dlt, #ApachePolaris, #ApacheSpark, and #Postgres. That spirit of learning, building, and supporting each other is what open source is all about ๐ฅ
Weโre still blown away by the community that came together for Women+ in OSS Day:
๐ ๐ญ๐ญ๐ฌ+ attendees
๐ ๐ญ๐ญ sponsors
๐ ๐ญ๐ฑ speakers
๐ ๐ต hands-on mentors
Huge thanks to our speakers for bringing so much thought, honesty, and energy to the day: @suehle, @HeatherMeeker4 , Krithika Balagurunathan, @lisancao, @nehapawar18, Tasha Drew, @holdenkarau, Stephanie Stattel, @rginn206, Stacey Haysler, India Walker-Smith, @gwenshap, @nyghtowl , and Lahari Chowtoori.
Our amazing sponsors for helping make the day possible: @Snowflake, @awscloud, @ClickHouseDB, @puppyquery, @fivetran, @groundcover_com, Stainless, @VeloDB_IO, @dltHub, @grafana, and @pgus.
And special appreciation to @TheDanicaFine, Zoe Steinkamp, @ZhenniWu, and Marcella dePunzio, whose time, care, and hard work helped bring it all together.
Photo credits: Evelyn Mietzner ๐
hashtag#WomenInOpenSource hashtag#OpenSource hashtag#WomenInTech hashtag#OSSCommunity hashtag#TechCommunity
Lisa N. Cao retweeted
Where is Apache Spark heading in Spark 4.2 and beyond? ๐
Join @lisancao and Apache Spark PMC member Hyukjin Kwon as they break down the migration from Python "Pickle" to Apache Arrow and whatโs next for the ecosystem.
๐Watch the full interview: youtu.be/zvq5UiVxpEg
Key Insights:
๐ฅ The Serialization Breakthrough: How Arrow solved the JVM-to-Python bottleneck
๐๏ธ The Zero-Copy Roadmap: Direct memory access coming in Spark 4.2/4.3
๐ ๏ธ Spark 4 Updates: New byte-count batching to prevent OOM issues
๐ก Expert Advice: When to enable Arrow for maximum performance
#ApacheSpark #ApacheArrow #Python
Lisa N. Cao retweeted
For a decade, PySpark developers have wrestled with a specific architectural tax: the overhead of moving data between the JVM and Python. It was the bottleneck that kept Python from feeling truly native to the Spark engine.
Then came Apache Arrow. ๐น
To celebrate 10 years of Arrow, weโre diving deep into the technical journey that transformed Spark into a zero-copy performance beast. In this video, @lisancao (@databricks), Matt Topol (Columnar), and Hyukjin Kwon (Databricks) map out exactly how this convergence happened and where itโs going in Spark 4.1. ๐
๐ฅ Watch the full 10-year retrospective here: youtu.be/EiEgU4m8XfM
They dive deep into:
๐ธ ๐๐ฒ๐๐ผ๐ป๐ฑ ๐ฃ๐ถ๐ฐ๐ธ๐น๐ฒ: Why the original serialization protocols couldn't scale and how Arrowโs columnar memory layout changed the game
๐ธ ๐ง๐ต๐ฒ ๐ฅ๐ผ๐ฎ๐ฑ๐บ๐ฎ๐ฝ: From the first integration in Spark 2.3 to the "Map in Arrow" API in 3.3
๐ธ ๐ง๐ต๐ฒ ๐๐ถ๐ป๐ฎ๐น ๐๐ผ๐ฟ๐บ: In Spark 4.1, Arrow isn't just a transport layer anymoreโitโs the native execution format
๐ธ ๐ง๐ต๐ฒ ๐ฃ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ ๐๐ถ๐ฒ๐ฟ๐ฎ๐ฟ๐ฐ๐ต๐: A definitive "Tier List" for 2026. Should you use Catalyst, Scala, or Arrow-Native UDFs?
#ApacheSpark #ApacheArrow #PySpark #OpenSource
Lisa N. Cao retweeted
Stop micromanaging your pipelines! ๐โ
Apache Spark 4.1 introduces ๐ฆ๐ฝ๐ฎ๐ฟ๐ธ ๐๐ฒ๐ฐ๐น๐ฎ๐ฟ๐ฎ๐๐ถ๐๐ฒ ๐ฃ๐ถ๐ฝ๐ฒ๐น๐ถ๐ป๐ฒ๐ (๐ฆ๐๐ฃ), and itโs a total shift in how we build data workflows. ๐
๐ง๐ต๐ฒ ๐ผ๐น๐ฑ ๐๐ฎ๐: Manual sequencing, retry logic, and dependency chaos. ๐ซ
๐ง๐ต๐ฒ ๐ฆ๐๐ฃ ๐๐ฎ๐: Declare your tables, define your flow, and let Spark do the heavy lifting. โ
Check out our latest video with a core SDP engineer, Sandy Ryza, to see how this feature automates:
โ๏ธ Parallelism
โ๏ธ Checkpointing
โ๏ธ Pipeline Sequencing
๐ฅ Watch the full talk: youtu.be/WNPYEZ7SMSM
#ApacheSpark #DataPipelines #SDP #DataEngineering #OpenSource
๐๐๐๐
Managing Spark transformations just got significantly easier with the introduction of ๐ฆ๐ฝ๐ฎ๐ฟ๐ธ ๐๐ฒ๐ฐ๐น๐ฎ๐ฟ๐ฎ๐๐ถ๐๐ฒ ๐ฃ๐ถ๐ฝ๐ฒ๐น๐ถ๐ป๐ฒ๐ (๐ฆ๐๐ฃ) in Apache Spark 4.1. ๐
As Lisa N. Cao breaks down in this clip, SDP handles the "heavy lifting" in the background so you don't have to. Here is what's happening under the hood:
๐น ๐๐ฒ๐ฝ๐ฒ๐ป๐ฑ๐ฒ๐ป๐ฐ๐ ๐ข๐ฟ๐ฑ๐ฒ๐ฟ๐ถ๐ป๐ด: SDP automatically detects table references and ensures your dependencies resolve before execution.
๐น ๐ฅ๐ฒ๐๐ถ๐น๐ถ๐ฒ๐ป๐ฐ๐ฒ: Built-in handling for checkpoints, parallelism, and retries.
๐น Language agnostic: Author your pipelines in Python or SQL, making your work more shareable and accessible.
๐น ๐ข๐ฟ๐ฐ๐ต๐ฒ๐๐๐ฟ๐ฎ๐๐ถ๐ผ๐ป: Works with Airflow to handle application-level tasks while SDP manages the Spark transformations internally.
Check out the new Spark Declarative Pipelines Programming Guide in the 4.1 docs to get started โก๏ธ spark.apache.org/docs/latestโฆ
#ApacheSpark #DeclarativePipelines #SparkConnect #OpenSource #PySpark #DataOps
Lisa N. Cao retweeted
Apache Spark 4.1 has officially landed and itโs a massive leap forward for productivity and performance! ๐
As Lisa Cao highlights in her latest update, this release is packed with features that fundamentally change how we build and manage Spark pipelines.
Key highlights include:
๐ ๐ฆ๐ฝ๐ฎ๐ฟ๐ธ ๐๐ฒ๐ฐ๐น๐ฎ๐ฟ๐ฎ๐๐ถ๐๐ฒ ๐ฃ๐ถ๐ฝ๐ฒ๐น๐ถ๐ป๐ฒ๐ (๐ฆ๐๐ฃ)
๐ ๐ฆ๐๐ฟ๐๐ฐ๐๐๐ฟ๐ฒ๐ฑ ๐ฆ๐๐ฟ๐ฒ๐ฎ๐บ๐ถ๐ป๐ด ๐ฅ๐ฒ๐ฎ๐น-๐ง๐ถ๐บ๐ฒ ๐ ๐ผ๐ฑ๐ฒ (๐ฅ๐ง๐ )
๐ ๐ฃ๐๐ฆ๐ฝ๐ฎ๐ฟ๐ธ ๐๐ป๐ต๐ฎ๐ป๐ฐ๐ฒ๐บ๐ฒ๐ป๐๐
๐ ๐ฆ๐ฝ๐ฎ๐ฟ๐ธ ๐๐ผ๐ป๐ป๐ฒ๐ฐ๐ & ๐ ๐
๐ ๐ฆ๐ค๐ ๐ฆ๐ฐ๐ฟ๐ถ๐ฝ๐๐ถ๐ป๐ด
๐ Check out the full release notes: spark.apache.org/releases/spโฆ
#ApacheSpark #SparkConnect #OpenSource #PySpark #DataOps
Lisa N. Cao retweeted
```/bin/bash``` just turned 38.
Happy birthday, bash!
To display bash shell version press `Ctrl+x Ctrl+v`
#unix #linux #opensource
Lisa N. Cao retweeted
Excited to be speaking about "The Failed Promises of Data Lineage: Why More Metadata Isnโt the Answer" on Oct 30th with @lisancao๐ธ ๐
Make sure to RSVP! ๐ luma.com/openlakehouse-1030
Lisa N. Cao retweeted
From building pipeline after pipeline to juggling data quality, CI/CD, and model deployments, one question keeps surfacing: What comes next?
Watch the Full episode with a special guest @lisancao
youtu.be/-rX4YOOlZ4M
Lisa N. Cao retweeted
From building pipeline after pipeline to juggling data quality, CI/CD, and model deployments, one question keeps surfacing: What comes next?
Watch the new episode with a special guest @lisancao
youtube.com/watch?v=-rX4YOOlโฆ
Lisa N. Cao retweeted
Thursday 6/5 (cont'd)
โฃ Mid-Career Reset: Design an energizing af work/life lu.ma/7xo31h3y
โฃ Daytona AI Builders โ Palo Alto, June '25 lu.ma/k0cckedc
@sambanovaai @ivanburazin
โฃ AI Demo-ish Day: A Vanta Community Show & Tell lu.ma/px73tsre
โฃ Startup Pitch Night lu.ma/rc2jawqn
@coltonkaplan
โฃ Data for AI Presents: The Open Source Afterparty w/ EQUINIX, Structured Labs, Apify, & more! lu.ma/p8xq3mcs
@lisancao
Lisa N. Cao retweeted
๐จ Early Bird ๐๐๐๐ ๐๐ฉ๐จ ๐๐ฃ๐ ๐๐ฃ 3 ๐ฟ๐๐ฎ๐จ! ๐จ
Get ready to meet the incredible ๐ @mgrygles ๐ @lisancao ๐ @techgirl1908 ๐ @holdenkarau ๐ @TheDanicaFine live at AI By the Bay 25 โladies who bring inspiration you wonโt find anywhere else
ai.bythebay.io/register
Loved this workshop! Check it out ;)
โจ Learn the Fundamentals of #DataOps in this workshop by Lisa N. Cao
Watch on YouTube: hubs.la/Q03hFmgV0
My episode on The Data Engineering Show is now live on YouTube, Spotify, Apple Podcasts, and Fame!
I had a blast with the @FireboltHQ guys and hope you enjoy. We got a bit excited and the fun is infectious! ๐ฅ
youtu.be/rAjMBiB8b8k?si=-Dqhโฆ
Lisa N. Cao retweeted
๐ง Why are devs loving #ApacheIceberg?
๐๏ธ In our latest Timeplus Live webinar, hear a panel discussion with @jove, @lisancao, and Ken Chen; and see Jove's demo for a C++ implementation of Iceberg read/write, now open-sourced by Timeplus
Full recording: youtube.com/watch?v=eCiSAFZuโฆ
If you haven't seen it, I was a guest on Karl's podcast last month. It was a really fun way to kick off the year after Data Day ๐
We talk about Apache Gravitino, DataOps, and future trends in data and AI architecture.
Available also on Apple + YT.
open.spotify.com/episode/3Qcโฆ
My "Fundamentals of DataOps" Workshop recording is now available on demand!
This workshop does deep dives into declarative programming, SRE practices, metadata, and K8s deployments in the Data Engineering Stack, geared for DevOps folk
@CDeliveryFdn
youtu.be/hZ8NpMgPKcM?si=kQvAโฆ
Lisa N. Cao retweeted
We're live @CDeliveryFdn workshop
DataOps infrastructures from scratch with @lisancao
Details: cd.foundation/workshops/ Join in for live Q&A.
























