@DataMechanics_

The Cloud-Native Spark Platform for Data Engineers. Run continuously optimized Apache Spark workloads on a managed kubernetes cluster in your cloud account.

datamechanics.co
Joined March 2020
  • Tweets63
  • Following433
  • Followers298
2Media
Since the Spark 3.2 release, the S3 Magic Committer is a lot more easy-to-use, performant, and stable. It enabled a 60% performance boost in our tests! Learn more in this technical blog post about #ApacheSpark performance tuning on #AWS #S3 spot.io/blog/improve-apache-…
2
4
Tutorial: Running PySpark inside Docker containers by @DataMechanics_ buff.ly/3pNjYSm
6
27
#ApacheSpark 3.2 was just released and it's now available on @DataMechanics_! New Features: * Spark-on-k8s specific: PVC reuse making Spark robust to exec loss by disaggregating compute & storage. * Spark now bundles Koalas, Hadoop 3.3.1, and RocksDB. datamechanics.co/blog-post/a…
9
6
Data Mechanics retweeted
Spot
@spot_flexera20 Oct 2021
Heard the big news? Spot by NetApp is excited to announce Spot #Security, a new product designed to keep your Cloud infrastructure secure. See how Spot helps you save 💰 in the cloud and protects you from cloud attacks. Read more 👉 hubs.ly/H0ZLkF90 #CloudOps @NetApp
2
4
Tutorial: Running PySpark inside Docker Containers: datamechanics.co/blog-post/t… A couple of months ago we published our Docker images for Apache Spark on DockerHub (they come packaged with common data connectors and libraries). This tutorial shows you how to get started with them!
3
5
Data Mechanics retweeted
AI by the Bay@ScaleByTheBay7 Oct 2021
.@JyStephan is a co-founder at @DataMechanics_ and really a people person. In our short conversation, he shared the joy of working within the team, with new clients and meeting his friends loom.ly/1iG7GtU
1
2
Replying to @DataMechanics_
@DataMechanics_ now powers the United Nations Global Platform, enabling statisticians around the word to analyse maritime traffic data. Migrating their #ApacheSpark platform from #EMR to #SparkOnKubernetes improved their UX & reduced their costs by 70%! datamechanics.co/blog-post/h…
3
1
Data Mechanics: Data Engineering with Jean-Yves Stephan @DataMechanics_ @JyStephan softwareengineeringdaily.com…

Data Mechanics: Data Engineering with Jean-Yves Stephan - Software Engineering Daily

Apache Spark is a popular open source analytics engine for large-scale data processing. Applications can be written in Java, Scala, Python, R, and SQL. These applications have flexible options to run...

softwareengineeringdaily.com
2
2
Data Mechanics retweeted
Jean-Yves Stephan@JyStephan8 May 2021
Humbled to be featured in the #DataEngineering Podcast: dataengineeringpodcast.com/d… We talked about how we try to make #ApacheSpark #CloudNative at @DataMechanics_ ! We talked about Delight, our open-source monitoring dashboard for Spark with CPU & Memory Metrics.. and much more!
3
9
.@DataMechanics_ introduces Delight, a cross-platform monitoring dashboard for Apache Spark helps you understand and improve the performance of your Spark applications buff.ly/33iEspu
6
13
This is the result of over 6 months of engineering work from our team - check it out and let us know your feedback!
Jean-Yves Stephan@JyStephan21 Apr 2021
You don't like the Spark UI? We heard you! We're releasing Delight - a free & cross-platform monitoring dashboard for #ApacheSpark. datamechanics.co/blog-post/d… Backed by an #opensource agent, It works on top of any Spark platform - Databricks, EMR, Dataproc, CDH/HDP, ...
3
2
Serverless #spark platforms sound interesting? We're stoked to be with @JyStephan the CEO of @DataMechanics_ tomorrow. towardsdatascience.com/how-w…
3
3
Data Mechanics retweeted
Kube Careers@kubecareers19 Apr 2021
💼 Solutions Architect with Data Mechanics @datamechanics_ Location: 🇺🇸🇨🇦🌍 USA/Canada/Remote 💰 150000 - 230000 USD per year Read the full job description → kube.careers
1
4
We're releasing our optimized #Docker images for #ApacheSpark at @DataMechanics_ ! Blog: datamechanics.co/blog-post/o… DockerHub: hub.docker.com/r/datamechani… They contain Spark, Scala, Java, Hadoop, Python, and connectors to common data sources. We hope to save you from dependency hell!
1
7
1
14
Spark on Kubernetes? We've got @JyStephan from @DataMechanics_ with us on Tuesday to tell us all about it. Sign up here- meetup.com/Data-on-Kubernete…
2
6
Data Mechanics retweeted
Jean-Yves Stephan@JyStephan6 Apr 2021
Excited to share this customer story of @Lingk_io who migrated from #EMR to our platform @DataMechanics_ The move to #SparkOnKubernetes enabled: * 65% AWS Costs reductions * 2x faster app startup time * 40% average app duration reduction More details: datamechanics.co/blog-post/m…
3
6
Data Mechanics retweeted
やっさん🍶
@yassan16830 Mar 2021
とうとうGAか。あとでちゃんと読もうっと。 Apache Spark 3.1 Release: Spark on Kubernetes is now Generally Available by @DataMechanics_ in @TDataScience towardsdatascience.com/apach…
8
15
.@DataMechanics_ goes over the main features of Spark 3.1, with a special focus on the improvements to Spark-on-Kubernetes. buff.ly/2PiTojS
9
13
Data Mechanics retweeted
Jean-Yves Stephan@JyStephan9 Mar 2021
Apache Spark 3.1 was released last week. In this release, the #Spark-on-#Kubernetes integration was officially declared Generally Available and production ready. What are the main features in 3.1? What's new with Spark-on-k8s? Read it here: datamechanics.co/blog-post/a…
8
15
Recording of last week's meetup on Our Successes & Challenges Running Spark on Kubernetes at Scale with @DataMechanics_ is out: youtube.com/watch?v=YjuZVbz2… It finishes with a live demo with jupyter notebooks, airflow integration, and custom docker image building. Thanks @Xomnia_BV
2
3
Load more