Open Source Azure Data Factory Alternatives

Kestra is an event-driven orchestration platform that unifies scheduled and real-time automation behind one declarative interface. Workflows are YAML, and the definition is rewritten whenever someone changes a flow from the UI, an API call or Terraform, so the orchestration logic stays managed as code no matter which door the change came through.

A flow lives in a namespace and holds tasks, while triggers decide when it runs, on a schedule or on an event such as a file arriving or a message landing in Kafka, Redis, Pulsar, MQTT, AWS SQS or Google Pub/Sub. Inputs and variables carry data through, and outputs generate artifacts visible in the interface.

The platform is language-agnostic, and the plugin ecosystem is where that shows.

• Run anywhere: Tasks execute locally, on remote servers over SSH, in Docker containers or as Kubernetes jobs.
• Any language: Python, Node.js, R, Go and Shell scripts run as tasks, alongside SQL queries and HTTP calls.
• Resilience: Retries, timeouts, error handling, conditional branching, backfills, subflows and dynamic tasks.
• Editor: Syntax highlighting, auto-completion, live validation and a topology view of the flow graph.
• Version control: Push flows from the built-in editor to a Git branch and keep CI/CD in the loop.

A single Docker command starts a local server on port 8080. Installation guides cover Docker Compose, Podman, Kubernetes, AWS, Google Cloud and Azure, with CloudFormation and Terraform templates for cloud deployments.

Airbyte moves data from where it is produced to where it gets analyzed. The catalog covers more than 600 connectors for APIs, databases, warehouses, lakes and AI applications, and that breadth is the argument: the long tail of sources is exactly what a closed integration tool never bothers to support.

When a connector does not exist yet, you build it. A no-code Connector Builder handles most sources and a low-code CDK covers the rest, and because connectors are open source you can fork and customize an existing one instead of filing a request and waiting.

The platform now splits along two jobs.

• ELT and ETL pipelines: the open source platform in this repository, or Airbyte Cloud, centralizes data into warehouses, lakes and databases.
• AI agent access: Airbyte Agents and the open source Agent SDK give LLMs and MCP clients real time reach into CRMs, support tools, SaaS APIs and databases.
• Typed connectors as tools: the SDK works with pydantic-ai, LangChain, OpenAI Agents and FastMCP, with retry, exception translation and output size guardrails included.
• Orchestration: syncs are triggered from Airflow, Dagster, Kestra or the Airbyte API rather than a scheduler of its own.
• Public registry: a generated registry report lists every connector available, and a hosted demo shows the platform before you deploy anything.

Deployment is either self-hosted from this repository or the managed cloud, with an enterprise tier adding further security features. Licensing is split between MIT and the Elastic License version 2, so check which one covers the components you intend to run.

Prefect is a workflow orchestration framework for building data pipelines in Python, aimed at the gap between a script that runs on someone's laptop and a pipeline the whole team depends on.

You add flow and task decorators to functions you have already written, and the framework takes over scheduling, retries, caching and observation. Pipelines can react to events, branch on conditions and recover from unexpected changes without being rewritten around a new abstraction.

A handful of pieces carry most of the work.

• Flows and tasks: two decorators turn plain Python functions into an orchestrated, tracked workflow.
• Deployments: calling serve with a cron expression and parameters puts a workflow on a schedule.
• Retries and caching: failures are retried and repeated work is cached without extra plumbing.
• Event automations: deployments can be triggered by events rather than only by the clock.
• Server and UI: one command runs a local Prefect server with a dashboard on port 4200.

Installation is pip or uv, and Prefect needs Python 3.10 or newer. Workflow activity can be monitored on a self-hosted Prefect server or on the managed Prefect Cloud dashboard, and runs can be started from the interface or the command line as well as from your own code.

Logstash is a data processing pipeline that runs on the server side, pulling from many sources simultaneously, transforming what arrives, and forwarding the result to wherever it should be stored. It replaces the collection of one-off scripts that usually accumulates between a log source and a search index.

It belongs to the Elastic Stack alongside Beats, Elasticsearch and Kibana, and Elasticsearch is the destination it is most often paired with. A pipeline is assembled from inputs, filters and outputs rather than written as transport code.

Extensibility is where most of the surface area lives.

• Over two hundred plugins: inputs, filters and outputs covering the sources and sinks most pipelines need.
• Self-contained gems: every plugin is a Ruby gem published to RubyGems and kept in its own repository under the logstash-plugins organization.
• Writing your own: the contributing guide covers developing and testing new plugins, which the project describes as straightforward.
• Separate issue tracking: plugin bugs and pull requests belong in that plugin's repository, while core work stays in this one.
• Packaged builds: officially released binaries plus Debian and RPM packages for the supported platforms.

Building from source needs JDK 21 with JAVA_HOME set, JRuby, rake and bundler, after which gradle tasks install the development and default gems. The repository holds both the Elastic-licensed X-Pack code and the OSS-licensed code, and an environment variable selects a build made only from the latter.

Mage OSS is a development environment for data pipelines that runs on your own machine. Writing an ETL job, wiring a transformation and watching it execute all happen in one notebook-style interface, with no cloud account required to begin.

Pipelines are built block by block, each block a piece of modular code in Python, SQL or R. The interactive editor doubles as the place to document what a block does, and dbt models can be built and run inside Mage rather than beside it.

The tooling around a pipeline is the part that usually costs the most time.

• Data integrations: prebuilt connectors to databases, APIs and cloud storage.
• Scheduling: jobs triggered by hand or on a schedule, with cron supported.
• Visual debugging: step-by-step execution with logs, live data previews and error handling.
• Quick setup: installation through Docker, pip or conda in a single command.
• Typical jobs: moving a spreadsheet into a warehouse, a daily SQL cleanup, or plain local ETL work.

Mage OSS covers local development and running pipelines on hardware you control. The project positions Mage Pro as the paid platform for teams that need multi-environment orchestration, role-based access control, monitoring and alerts, and CI/CD, available managed, hybrid or on-premises. Setup instructions and guides live in the project documentation.