OSS for science, #python, @pangeo_data, @xarray_dev team. Ex-fusion, oceanography, now @_cworthy 🦋 https://xitter.cf/t.co/4jprMBeckr
New York
Joined February 2020
- Tweets501
- Following680
- Followers646
- Likes1.8K
Tom Nicholas retweeted
Science needs a social network for sharing big data hackmd.io/@TomNicholas/H1Kzo… by @TEGNicholasCode
Tom Nicholas retweeted
We're moving over to BlueSky and LinkedIn for all our future announcements. Follow us at bsky.app/profile/pangeo.io to find out more about tomorrow's showcase 😉 (p.s., it's on Xpublish at Scale at 4 PM EST 🚀) Connect with us on LinkedIn at linkedin.com/company/pangeo-…
Tom Nicholas retweeted
Our friend's over at @zarr_dev made a big release today!
Xarray v2025.01.1 was also released today with full support for Zarr-Python 3 🚀
🎉 Zarr-Python 3 is here! 🎉
- Full support for Zarr v3 spec
- Chunk-sharding for more efficient data storage
- Major performance boosts with async I/O & parallel compression
💻 pip install --upgrade zarr
Blog post: zarr.dev/blog/zarr-python-3-…
Tom Nicholas retweeted
🌤️ #AMS2025 is just around the corner! We are taking AMS by storm with an exhibitor booth (booth 353), two talks from @_jhamman and @rabernat , and hosting a @pangeo_data Community Happy Hour (register here: lu.ma/ddtba5f5)!
Tom Nicholas retweeted
That said, it isn't 100% clear that NASA's best move is to immediately convert 10000+ data sets into cutting edge ARCO formats. Kerchunk and Virtual Zarr offer benefits of ARCO while keeping data in the native formats.
Tom Nicholas retweeted
Come learn about recent @xarray_dev GroupBy improvements at tomorrow's (Wed, Nov 13) Pangeo Showcase!
discourse.pangeo.io/t/pangeo…
Tom Nicholas retweeted
We've talked a lot about #Icechunk's performance this week 🚀. But the Zarr-Python 3 results are also very encouraging! We're a few weeks away from the 3.0 launch but what this chart shows is that the new AsyncIO + multi-threading functionality in Zarr is going to be really good.
ALSO this release is the first to be compatible with the much anticipated v3 implementation of zarr-python! (still on its beta branch right now)
This brings big performance benefits when reading @zarr_dev on S3 via async and (b) compatibility with @EarthmoverHQ 's Icechunk.
Tom Nicholas retweeted
⚡️ Icechunk is fast! What does this mean for users? Reduced cost for all data-intensive compute jobs and enhanced productivity for the data scientists who work with data all day long.
Icechunk, @EarthmoverHQ's new transactional cloud-native storage engine for array / tensor data, works together with @zarr_dev , augmenting the Zarr core data model with features that enhance performance, collaboration, and safety in a multi-user cloud-computing context.
Reading data through Icechunk is 36x faster than trying to read HDF5 files from cloud object storage, 6x faster than regular Zarr alone, and 2.5x faster than regular Zarr + Dask. Most importantly, Icechunk can achieve throughput on par with the compute instance network bandwidth, the "hardware limit" for I/O bound workloads.
Want to learn more about this benchmark? Come to our Icechunk informational webinar tomorrow, Tuesday, October 22nd from 12 - 1 PM EST. Registration link: share.hsforms.com/1SCOFqe2kT…
ALT Performance analysis of various I/O stacks for reading data from S3. The NetCDF dataset was a 1.8 GB NetCDF file (4GB uncompressed) from the NSF NCAR Curated ECMWF Reanalysis 5 (ERA5) The dataset was transformed to Zarr with zstd compression and written in both V2, V3, and V3 + Icechunk format. It was read back using Xarray plus different I/O stacks, with and without Dask.
Tom Nicholas retweeted
🎉 @source_coop is now open source!
The web application - github.com/source-cooperativ… - and the data proxy - github.com/source-cooperativ… - have been opened up & updated with documentation on how to get it running locally.
More documentation coming soon + tasks for new developers!
Tom Nicholas retweeted
We’re hosting a webinar on Tuesday, October 22 from 12- 1 PM EST to discuss what Icechunk means for the scientific data community and answer questions from attendees. Register here: share.hsforms.com/1SCOFqe2kT…
🚀 We are thrilled to announce the release of the Icechunk storage engine, a new open-source library and specification for the storage of multidimensional array (a.k.a. tensor) data in cloud object storage.
Read our blog post about Icechunk here: earthmover.io/blog/icechunk
Tom Nicholas retweeted
Great opportunity to work with @BalwadaDhruv, one of most innovative physical oceanographers in the world, at @LamontEarth in NYC!
We are looking to hire a postdoctoral scholar at Lamont Doherty Earth Observatory to work on submesoscale and mesoscale ocean turbulence using observations and machine learning: academic.careers.columbia.ed…
Tom Nicholas retweeted
Not all ocean alkalinity enhancement (OAE) deployments will work the same because not all parts of the ocean work the same. We built a new interactive mapping tool using data from @_CWorthy to show how OAE changes based on where and when it’s deployed.
Tom Nicholas retweeted
Earthmover is a quietly transformative company building at the vanguard of a movement that is starting in elite research institutions but will explode into commercial and government markets as people realize the power of what they are building and its peerlessness.
Tom Nicholas retweeted
Icechunk's design solves so many of the headaches we faced when trying to update Zarr datasets on the Planetary Computer. Check it out!
🚀 We are thrilled to announce the release of the Icechunk storage engine, a new open-source library and specification for the storage of multidimensional array (a.k.a. tensor) data in cloud object storage.
Read our blog post about Icechunk here: earthmover.io/blog/icechunk
Tom Nicholas retweeted
We just dropped something big @EarthmoverHQ - a new transactional storage engine for @zarr_dev.
🚀 We are thrilled to announce the release of the Icechunk storage engine, a new open-source library and specification for the storage of multidimensional array (a.k.a. tensor) data in cloud object storage.
Read our blog post about Icechunk here: earthmover.io/blog/icechunk
Tom Nicholas retweeted
🚀 We are thrilled to announce the release of the Icechunk storage engine, a new open-source library and specification for the storage of multidimensional array (a.k.a. tensor) data in cloud object storage.
Read our blog post about Icechunk here: earthmover.io/blog/icechunk
Tom Nicholas retweeted
📢 Join us for the Zarr-Python V3 documentation sprint from 30th September to 1st October! 📝
Help us write and shape the V3 docs from the ground up. It's a great chance to contribute to Zarr-Python's future!
More details here: github.com/zarr-developers/z…
Tom Nicholas retweeted
At long last Xarray can now group by multiple arrays. The culmination of 3 years of on/off work
xarray.dev/blog/multiple-gro…
Try it!
Tom Nicholas retweeted
We're to build a 100-TB scale geospatial benchmark suite
docs.coiled.io/blog/geospati…
We've seen an uptick in geospatial users and in challenges of the Xarray/Dask stack to scale beyond ~500-GiB.
This post presents a call for benchmark workloads.
Tom Nicholas retweeted
pretty interesting point from podcast with @bernhardsson that unit tests running in sequence or the concept of clusters are skeuomorphic abstractions in cloud world.
no reason why can't just have your tests run in parallel, we have to re-orient the way we think!