The Role of Airflow in Building Smarter ML Pipelines at Vivian Health with Max Calehuff

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

The integration of data orchestration and machine learning is critical to operational efficiency in healthcare tech. Vivian Health leverages Airflow to power both its ETL pipelines and ML workflows while maintaining strict compliance standards.

Max Calehuff, Lead Data Engineer at Vivian Health, joins us to discuss how his team uses Airflow for ML ops, regulatory compliance and large-scale data orchestration. He also shares insights into upgrading to Airflow 3 and the importance of balancing flexibility with security in a healthcare environment.

Key Takeaways:

00:00 Introduction.

04:21 The role of Airflow in managing ETL pipelines and ML retraining.

06:23 Using AWS SageMaker for ML training and deployment.

07:47 Why Airflow’s versatility makes it ideal for MLOps.

10:50 The importance of documentation and best practices for engineering teams.

13:44 Automating anonymization of user data for compliance.

15:30 The benefits of remote execution in Airflow 3 for regulated industries.

18:16 Quality-of-life improvements and desired features in future Airflow versions.

Resources Mentioned:

Max Calehuff

https://www.linkedin.com/in/maxwell-calehuff/

Vivian Health | LinkedIn

https://www.linkedin.com/company/vivianhealth/

Vivian Health | Website

https://www.vivian.com

Apache Airflow

https://airflow.apache.org/

Astronomer

https://www.astronomer.io/

AWS SageMaker

https://www.google.com/aclk?sa=L&ai=DChsSEwj3-fbz1tiQAxWXlKYDHXUBBVoYACICCAEQABoCdGI&ae=2&aspm=1&co=1&ase=2&gclid=Cj0KCQiA5abIBhCaARIsAM3-zFWbfj2olUvX4dqoiYNaE3q2fMf_ZifRjmbKNQCVX7D6ZMClaUXUkFkaAuwmEALw_wcB&cid=CAASQuRoMccxWhBvMq-1Uez3XOZti1ul7mTDotKvSMoDHv0q2xCsyS2FzMptO5dJf3tmfkLRu22TtD8ChTmdjvs6YetTjQ&cce=2&category=acrcp_v1_35&sig=AOD64_2xE2xolEEVbpDb56qXQluxTzs-Aw&q&nis=4&adurl&ved=2ahUKEwj7le3z1tiQAxWXcvUHHfZePbAQ0Qx6BAgUEAE

dbtLabs

https://www.getdbt.com/

Cosmos

https://github.com/astronomer/astronomer-cosmos

Split

https://www.split.io/

Snowflake

https://www.snowflake.com/en/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow