Bacalhau

Shipping petabytes across a network to reach a compute cluster is slow, expensive, and drags sensitive data out of the boundary it was meant to stay inside. Bacalhau inverts the pattern by sending the job to the data instead, executing work close to where it was created.

Everything ships as a single self-contained binary that acts as client, orchestrator, or compute node depending on how you start it. An orchestrator coordinates scheduling while compute nodes run the tasks, and execution engines and storage providers sit behind clean interfaces, so Docker and WebAssembly workloads both fit. Jobs are declared in YAML or passed as command line arguments.

Several consequences follow from that model.

• Parallel by default: jobs process at the source and run in parallel without extra configuration.
• Lower transfer cost: keeping compute near the data reduces or removes ingress and egress charges.
• Data sovereignty: sensitive material is processed within your own security boundary, and scrubbing can happen before any migration.
• Storage integrations: S3, HTTP and HTTPS endpoints, IPFS, and local storage are supported, and results publish to local volumes, S3, or other backends.
• Job types: batch, ops, daemon, and service jobs cover different workflow shapes.
• Cross-organizational compute: vetted computations run against protected datasets without exposing the raw data.

The CLI installs with one shell command on Linux and macOS. Uses named by the project include log processing at scale, distributed data warehousing, fleet management, distributed machine learning, and edge computing. The code is Apache 2.0, with the name and logos reserved under trademark terms.