Pandas groupby is a powerful function that groups distinct sets within selected columns and aggregates metrics from other columns accordingly. Performing these operations results in a pivot table, something that’s very useful in data analysis. Aggregating Multiple Columns In this article, I share a technique for computing ad-hoc aggregations that can involve multiple…

Personal knowledge graphs are rapidly growing in popularity as benefits emerge. There are lots to chose from, but here’s why I love Obsidian. Building my own knowledge graph Roam Research was the first tool like this that I learned about — their revolutionary graph approach to note taking blew my mind a bit. I had…

Databases are like Pokémon. Gotta pass data between em’ all! There are many methods of sending data to and from BigQuery. Some are even documented! In this post I’ll share a simple method that I use to copy tables from Redshift into BigQuery using only SQL and command line tools. Data Synchronization …

In Python it’s okay to make assumptions, as long as you’re able to clean up the mess if they turn out to be wrong. In fact, this is not only okay but considered good practice. Real life example: dogs & cats This section is for people who want to learn about the basic try/except control structure…

In this post, I introduce the concept of dynamic DAG creation and explain the significance of Python global variables for Airflow. What do I mean by “dynamic DAG”? Dynamic DAG creation is important for scalable data pipeline applications. When confined to the realm of static DAG scripts, we find ourselves duplicating code in order to create pipelines. …

Shortcuts 🥱 Symbolic links are handy for shortcuts in your file explorer. Maybe you have shortcuts on your desktop to your favorite folders, or maybe you symlink your active projects in your home directory for quick bash access. For example: ln -s /User/alex/Apps/2020/appthing /User/alex/appthing This will create a shortcut to ~/Apps/2020/appthing directly…

If you’re like me then you try to avoid using the mouse whenever possible. This post will help with that. The Action to Automate My “normal” workflow involves opening new application windows pretty frequently. For example, I might open a new terminal window to start working on a project, or I might open a…

Last night I fell down the rabbit hole of different ways to configure docker apps with runtime arguments. Somehow it ended with me searching for ASCII art of a d20.. I settled on this: If you’re thinking “what does ASCII art of dice have to do with docker?” Nothing really…

Why you should think about using JSON Line format in your data processing workflow. We’ll look at some jsonl examples and discuss how I use it day to day. What is the json line format? It’s a file type specification where each line is a JSON object. Just imagine a bunch of stacked up dictionaries. …