Lisa N. Cao (@lisancao) header
Lisa N. Cao (@lisancao) profile picture

DevRel at Databricks. I talk data engineering and AI. Bluesky: https://nitter.gq/t.co/LBgVMpvu4D

๐ŸŒŽ, Milky Way Galaxy
Joined February 2011
Photo by Lisa N. Cao (@lisancao)Photo by Lisa N. Cao (@lisancao)Photo by Lisa N. Cao (@lisancao)Photo by Lisa N. Cao (@lisancao)Photo by Lisa N. Cao (@lisancao)Photo by Lisa N. Cao (@lisancao)Photo by Lisa N. Cao (@lisancao)Photo by Lisa N. Cao (@lisancao)Photo by Lisa N. Cao (@lisancao)Photo by Lisa N. Cao (@lisancao)
I've pretty much been over at Bluesky, catch you there ;) bsky.app/profile/lisancao.bsโ€ฆ
489
Lisa N. Cao retweeted
Sometimes the most important step is simply ๐˜€๐—ต๐—ผ๐˜„๐—ถ๐—ป๐—ด ๐˜‚๐—ฝ. Women+ in OSS Day on Monday was a great reminder of that ๐Ÿ’œ Open source can feel intimidating from the outside, but this event made one thing clear: every contribution ๐—บ๐—ฎ๐˜๐˜๐—ฒ๐—ฟ๐˜€. Code, documentation, testing, reviews, questions, or simply helping someone get started all count. Beyond the talks, people rolled up their sleeves and contributed to OSS projects like #dlt, #ApachePolaris, #ApacheSpark, and #Postgres. That spirit of learning, building, and supporting each other is what open source is all about ๐Ÿ”ฅ Weโ€™re still blown away by the community that came together for Women+ in OSS Day: ๐Ÿ’œ ๐Ÿญ๐Ÿญ๐Ÿฌ+ attendees ๐Ÿ’œ ๐Ÿญ๐Ÿญ sponsors ๐Ÿ’œ ๐Ÿญ๐Ÿฑ speakers ๐Ÿ’œ ๐Ÿต hands-on mentors Huge thanks to our speakers for bringing so much thought, honesty, and energy to the day: @suehle, @HeatherMeeker4 , Krithika Balagurunathan, @lisancao, @nehapawar18, Tasha Drew, @holdenkarau, Stephanie Stattel, @rginn206, Stacey Haysler, India Walker-Smith, @gwenshap, @nyghtowl , and Lahari Chowtoori. Our amazing sponsors for helping make the day possible: @Snowflake, @awscloud, @ClickHouseDB, @puppyquery, @fivetran, @groundcover_com, Stainless, @VeloDB_IO, @dltHub, @grafana, and @pgus. And special appreciation to @TheDanicaFine, Zoe Steinkamp, @ZhenniWu, and Marcella dePunzio, whose time, care, and hard work helped bring it all together. Photo credits: Evelyn Mietzner ๐ŸŽ‰ hashtag#WomenInOpenSource hashtag#OpenSource hashtag#WomenInTech hashtag#OSSCommunity hashtag#TechCommunity
1
6
164
Lisa N. Cao retweeted
Where is Apache Spark heading in Spark 4.2 and beyond? ๐Ÿš€ Join @lisancao and Apache Spark PMC member Hyukjin Kwon as they break down the migration from Python "Pickle" to Apache Arrow and whatโ€™s next for the ecosystem. ๐Ÿ”—Watch the full interview: youtu.be/zvq5UiVxpEg Key Insights: ๐Ÿ”ฅ The Serialization Breakthrough: How Arrow solved the JVM-to-Python bottleneck ๐ŸŽ๏ธ The Zero-Copy Roadmap: Direct memory access coming in Spark 4.2/4.3 ๐Ÿ› ๏ธ Spark 4 Updates: New byte-count batching to prevent OOM issues ๐Ÿ’ก Expert Advice: When to enable Arrow for maximum performance #ApacheSpark #ApacheArrow #Python
1
3
23
1,411
Lisa N. Cao retweeted
For a decade, PySpark developers have wrestled with a specific architectural tax: the overhead of moving data between the JVM and Python. It was the bottleneck that kept Python from feeling truly native to the Spark engine. Then came Apache Arrow. ๐Ÿน To celebrate 10 years of Arrow, weโ€™re diving deep into the technical journey that transformed Spark into a zero-copy performance beast. In this video, @lisancao (@databricks), Matt Topol (Columnar), and Hyukjin Kwon (Databricks) map out exactly how this convergence happened and where itโ€™s going in Spark 4.1. ๐Ÿš€ ๐ŸŽฅ Watch the full 10-year retrospective here: youtu.be/EiEgU4m8XfM They dive deep into: ๐Ÿ”ธ ๐—•๐—ฒ๐˜†๐—ผ๐—ป๐—ฑ ๐—ฃ๐—ถ๐—ฐ๐—ธ๐—น๐—ฒ: Why the original serialization protocols couldn't scale and how Arrowโ€™s columnar memory layout changed the game ๐Ÿ”ธ ๐—ง๐—ต๐—ฒ ๐—ฅ๐—ผ๐—ฎ๐—ฑ๐—บ๐—ฎ๐—ฝ: From the first integration in Spark 2.3 to the "Map in Arrow" API in 3.3 ๐Ÿ”ธ ๐—ง๐—ต๐—ฒ ๐—™๐—ถ๐—ป๐—ฎ๐—น ๐—™๐—ผ๐—ฟ๐—บ: In Spark 4.1, Arrow isn't just a transport layer anymoreโ€”itโ€™s the native execution format ๐Ÿ”ธ ๐—ง๐—ต๐—ฒ ๐—ฃ๐—ฒ๐—ฟ๐—ณ๐—ผ๐—ฟ๐—บ๐—ฎ๐—ป๐—ฐ๐—ฒ ๐—›๐—ถ๐—ฒ๐—ฟ๐—ฎ๐—ฟ๐—ฐ๐—ต๐˜†: A definitive "Tier List" for 2026. Should you use Catalyst, Scala, or Arrow-Native UDFs? #ApacheSpark #ApacheArrow #PySpark #OpenSource
1
6
61
3,521
Lisa N. Cao retweeted
Stop micromanaging your pipelines! ๐Ÿ›‘โœ‹ Apache Spark 4.1 introduces ๐—ฆ๐—ฝ๐—ฎ๐—ฟ๐—ธ ๐——๐—ฒ๐—ฐ๐—น๐—ฎ๐—ฟ๐—ฎ๐˜๐—ถ๐˜ƒ๐—ฒ ๐—ฃ๐—ถ๐—ฝ๐—ฒ๐—น๐—ถ๐—ป๐—ฒ๐˜€ (๐—ฆ๐——๐—ฃ), and itโ€™s a total shift in how we build data workflows. ๐Ÿ”„ ๐—ง๐—ต๐—ฒ ๐—ผ๐—น๐—ฑ ๐˜„๐—ฎ๐˜†: Manual sequencing, retry logic, and dependency chaos. ๐Ÿ˜ซ ๐—ง๐—ต๐—ฒ ๐—ฆ๐——๐—ฃ ๐˜„๐—ฎ๐˜†: Declare your tables, define your flow, and let Spark do the heavy lifting. โœ… Check out our latest video with a core SDP engineer, Sandy Ryza, to see how this feature automates: โš™๏ธ Parallelism โš™๏ธ Checkpointing โš™๏ธ Pipeline Sequencing ๐ŸŽฅ Watch the full talk: youtu.be/WNPYEZ7SMSM #ApacheSpark #DataPipelines #SDP #DataEngineering #OpenSource
3
33
1
208
12,301
๐Ÿ’–๐Ÿ’–๐Ÿ’–๐Ÿ’–
Managing Spark transformations just got significantly easier with the introduction of ๐—ฆ๐—ฝ๐—ฎ๐—ฟ๐—ธ ๐——๐—ฒ๐—ฐ๐—น๐—ฎ๐—ฟ๐—ฎ๐˜๐—ถ๐˜ƒ๐—ฒ ๐—ฃ๐—ถ๐—ฝ๐—ฒ๐—น๐—ถ๐—ป๐—ฒ๐˜€ (๐—ฆ๐——๐—ฃ) in Apache Spark 4.1. ๐Ÿš€ As Lisa N. Cao breaks down in this clip, SDP handles the "heavy lifting" in the background so you don't have to. Here is what's happening under the hood: ๐Ÿ”น ๐——๐—ฒ๐—ฝ๐—ฒ๐—ป๐—ฑ๐—ฒ๐—ป๐—ฐ๐˜† ๐—ข๐—ฟ๐—ฑ๐—ฒ๐—ฟ๐—ถ๐—ป๐—ด: SDP automatically detects table references and ensures your dependencies resolve before execution. ๐Ÿ”น ๐—ฅ๐—ฒ๐˜€๐—ถ๐—น๐—ถ๐—ฒ๐—ป๐—ฐ๐—ฒ: Built-in handling for checkpoints, parallelism, and retries. ๐Ÿ”น Language agnostic: Author your pipelines in Python or SQL, making your work more shareable and accessible. ๐Ÿ”น ๐—ข๐—ฟ๐—ฐ๐—ต๐—ฒ๐˜€๐˜๐—ฟ๐—ฎ๐˜๐—ถ๐—ผ๐—ป: Works with Airflow to handle application-level tasks while SDP manages the Spark transformations internally. Check out the new Spark Declarative Pipelines Programming Guide in the 4.1 docs to get started โžก๏ธ spark.apache.org/docs/latestโ€ฆ #ApacheSpark #DeclarativePipelines #SparkConnect #OpenSource #PySpark #DataOps
97
Lisa N. Cao retweeted
Apache Spark 4.1 has officially landed and itโ€™s a massive leap forward for productivity and performance! ๐Ÿš€ As Lisa Cao highlights in her latest update, this release is packed with features that fundamentally change how we build and manage Spark pipelines. Key highlights include: ๐ŸŒŸ ๐—ฆ๐—ฝ๐—ฎ๐—ฟ๐—ธ ๐——๐—ฒ๐—ฐ๐—น๐—ฎ๐—ฟ๐—ฎ๐˜๐—ถ๐˜ƒ๐—ฒ ๐—ฃ๐—ถ๐—ฝ๐—ฒ๐—น๐—ถ๐—ป๐—ฒ๐˜€ (๐—ฆ๐——๐—ฃ) ๐ŸŒŸ ๐—ฆ๐˜๐—ฟ๐˜‚๐—ฐ๐˜๐˜‚๐—ฟ๐—ฒ๐—ฑ ๐—ฆ๐˜๐—ฟ๐—ฒ๐—ฎ๐—บ๐—ถ๐—ป๐—ด ๐—ฅ๐—ฒ๐—ฎ๐—น-๐—ง๐—ถ๐—บ๐—ฒ ๐— ๐—ผ๐—ฑ๐—ฒ (๐—ฅ๐—ง๐— ) ๐ŸŒŸ ๐—ฃ๐˜†๐—ฆ๐—ฝ๐—ฎ๐—ฟ๐—ธ ๐—˜๐—ป๐—ต๐—ฎ๐—ป๐—ฐ๐—ฒ๐—บ๐—ฒ๐—ป๐˜๐˜€ ๐ŸŒŸ ๐—ฆ๐—ฝ๐—ฎ๐—ฟ๐—ธ ๐—–๐—ผ๐—ป๐—ป๐—ฒ๐—ฐ๐˜ & ๐— ๐—Ÿ ๐ŸŒŸ ๐—ฆ๐—ค๐—Ÿ ๐—ฆ๐—ฐ๐—ฟ๐—ถ๐—ฝ๐˜๐—ถ๐—ป๐—ด ๐Ÿ”— Check out the full release notes: spark.apache.org/releases/spโ€ฆ #ApacheSpark #SparkConnect #OpenSource #PySpark #DataOps
2
9
1
111
6,466
```/bin/bash``` just turned 38. Happy birthday, bash! To display bash shell version press `Ctrl+x Ctrl+v` #unix #linux #opensource
9
68
6
355
21,056
From building pipeline after pipeline to juggling data quality, CI/CD, and model deployments, one question keeps surfacing: What comes next? Watch the Full episode with a special guest @lisancao youtu.be/-rX4YOOlZ4M
1
3
173
From building pipeline after pipeline to juggling data quality, CI/CD, and model deployments, one question keeps surfacing: What comes next? Watch the new episode with a special guest @lisancao youtube.com/watch?v=-rX4YOOlโ€ฆ
2
5
122
Thursday 6/5 (cont'd) โ€ฃ Mid-Career Reset: Design an energizing af work/life lu.ma/7xo31h3y โ€ฃ Daytona AI Builders โ€“ Palo Alto, June '25 lu.ma/k0cckedc @sambanovaai @ivanburazin โ€ฃ AI Demo-ish Day: A Vanta Community Show & Tell lu.ma/px73tsre โ€ฃ Startup Pitch Night lu.ma/rc2jawqn @coltonkaplan โ€ฃ Data for AI Presents: The Open Source Afterparty w/ EQUINIX, Structured Labs, Apify, & more! lu.ma/p8xq3mcs @lisancao
2
1
2
860
๐Ÿšจ Early Bird ๐™๐™ž๐™˜๐™ ๐™š๐™ฉ๐™จ ๐™€๐™ฃ๐™™ ๐™ž๐™ฃ 3 ๐˜ฟ๐™–๐™ฎ๐™จ! ๐Ÿšจ Get ready to meet the incredible ๐Ÿ’— @mgrygles ๐Ÿ’— @lisancao ๐Ÿ’— @techgirl1908 ๐Ÿ’— @holdenkarau ๐Ÿ’— @TheDanicaFine live at AI By the Bay 25 โ€”ladies who bring inspiration you wonโ€™t find anywhere else ai.bythebay.io/register
4
1
4
1,010
We had some amazing abstracts this year :) see you in June!
84
Loved this workshop! Check it out ;)
โœจ Learn the Fundamentals of #DataOps in this workshop by Lisa N. Cao Watch on YouTube: hubs.la/Q03hFmgV0
2
101
My episode on The Data Engineering Show is now live on YouTube, Spotify, Apple Podcasts, and Fame! I had a blast with the @FireboltHQ guys and hope you enjoy. We got a bit excited and the fun is infectious! ๐Ÿ”ฅ youtu.be/rAjMBiB8b8k?si=-Dqhโ€ฆ
173
Lisa N. Cao retweeted
๐ŸงŠ Why are devs loving #ApacheIceberg? ๐ŸŽ™๏ธ In our latest Timeplus Live webinar, hear a panel discussion with @jove, @lisancao, and Ken Chen; and see Jove's demo for a C++ implementation of Iceberg read/write, now open-sourced by Timeplus Full recording: youtube.com/watch?v=eCiSAFZuโ€ฆ
1
1
164
If you haven't seen it, I was a guest on Karl's podcast last month. It was a really fun way to kick off the year after Data Day ๐Ÿ’– We talk about Apache Gravitino, DataOps, and future trends in data and AI architecture. Available also on Apple + YT. open.spotify.com/episode/3Qcโ€ฆ
74
My "Fundamentals of DataOps" Workshop recording is now available on demand! This workshop does deep dives into declarative programming, SRE practices, metadata, and K8s deployments in the Data Engineering Stack, geared for DevOps folk @CDeliveryFdn youtu.be/hZ8NpMgPKcM?si=kQvAโ€ฆ
1
2
4
445
Also huge thanks to @cloudnativeboy for hosting me!
1
59
We're live @CDeliveryFdn workshop DataOps infrastructures from scratch with @lisancao Details: cd.foundation/workshops/ Join in for live Q&A.
2
3
281