Moving Externally Orchestrated Prefect Pipelines to Prefect Cloud
One of the things that makes Prefect such a great data pipeline framework (in my opinion) is the ability to write a pipeline as a standalone piece of code, a...
Home
Dev Log
Recent Posts
Remote Spark Execution Tasks in Prefect Flows using Databricks Connect
Continuing on our pipeline from Prefect Local Pipeline, one of the more stick pieces of our pipeline is the fact that we use spark_submit on a local machine ...
Prefect Local Pipeline
Orchestration: Local Compute: Local Storage: Cloud
Bridge & Tunnel Data 4 Ways
Here we look at a few different ways to store the MTA Bridge and Tunnel Hourly Traffic Dataset for further analysis.