I was talking with my friend @nealriley about how much fun I’m having with LLMs lately, including the project I've written about before to interpret screenshots of podcasts and YouTube videos. I made the observation that working with LLMs seems to boil down to a pattern of creating ETL pipelines, over and over again. He quickly corrected me, suggesting, "Don't you mean ELT pipelines?" I laughed at his curious and unexpected comment, which seemed unusually pedantic. However, I found myself thinking about his comment for days afterward. It reminded me of a quip from Rich Hickey, inventor of the Clojure programming language: "Information is simple. The only thing you can do with information is ruin it." In my experience, what Rich said is so true, and shows that Neal was absolutely correct. What I meant to say is "ELT," not "ETL." In other words, you should Extract, Load/Store, and then Transform data, in that order (ELT). Unless you love misery, you should not Transform before you Load/Store (ETL). Over the last many decades, I've ruined data in so many many ways, for me, the “T” in “ETL” stands for tarnish, trash, taint, etc. I’ve accidentally converted date-time strings incorrectly (turning them all into nils), I’ve discarded data fields that I thought were irrelevant, but years later, I wish I had kept them because it is priceless information, impossible to reconstruct. I’ve accidentally overwritten data, deleted data, and so many more terrible things, Thankfully, I've wiped those episodes from my memory, lest I start kicking myself again. This is because I was Transforming the data (or Trashing it) before storing it. One of the things I've learned from the Clojure and cloud community is the value of just storing data in its original form, often in storage buckets, deferring any transformation of the data to afterward, often at runtime (i.e., Extract, Load/Store, then Transform). I almost made this mistake last week: I mentioned how I use @revai and @deepgram and to create podcast audio transcripts. At first, I converted the audio transcripts to match the format used by the python-transcript-api program. That way, I could use the same code to render the transcript, regardless of what the transcript source was. This is when I once again learned that the only thing I could do to the data is to ruin it. So instead, I saved the entirety of the transcript in its original format, including the full API return payload. That way, I’m preserving optionality, enabling myself to use that data in new ways in the future. By discarding the data, I destroy that optionality. Today, I saw one such potential option: I realized that transcript diarization could be helpful (this is when transcripts detect when there are multiple speakers talking, and ideally labels them). The YouTube-transcription-api doesn’t support speaker labels, and I would have discarded that data to match it. My lessons: - Store your data in as close to its original form - Defer data transformation to after you load/store it