Prefect

Prefect is a workflow orchestration framework for building data pipelines in Python, aimed at the gap between a script that runs on someone's laptop and a pipeline the whole team depends on.

You add flow and task decorators to functions you have already written, and the framework takes over scheduling, retries, caching and observation. Pipelines can react to events, branch on conditions and recover from unexpected changes without being rewritten around a new abstraction.

A handful of pieces carry most of the work.

• Flows and tasks: two decorators turn plain Python functions into an orchestrated, tracked workflow.
• Deployments: calling serve with a cron expression and parameters puts a workflow on a schedule.
• Retries and caching: failures are retried and repeated work is cached without extra plumbing.
• Event automations: deployments can be triggered by events rather than only by the clock.
• Server and UI: one command runs a local Prefect server with a dashboard on port 4200.

Installation is pip or uv, and Prefect needs Python 3.10 or newer. Workflow activity can be monitored on a self-hosted Prefect server or on the managed Prefect Cloud dashboard, and runs can be started from the interface or the command line as well as from your own code.