Building and Testing a new Apache Airflow Plugin
Recently, I had the opportunity to add a new EMR on EKS plugin to Apache Airflow. While I’ve been a consumer of Airflow over the years, I’ve never contributed directly to the project. And weighing in at over half a million lines of code, Airflow is a pretty complex project to wade into. So here’s a guide on how I made a new operator in the AWS provider package. Overview Before you get started, it’s good to have an understanding of the different components of an Airflow task....