Logstash

Logstash is a data processing pipeline that runs on the server side, pulling from many sources simultaneously, transforming what arrives, and forwarding the result to wherever it should be stored. It replaces the collection of one-off scripts that usually accumulates between a log source and a search index.

It belongs to the Elastic Stack alongside Beats, Elasticsearch and Kibana, and Elasticsearch is the destination it is most often paired with. A pipeline is assembled from inputs, filters and outputs rather than written as transport code.

Extensibility is where most of the surface area lives.

• Over two hundred plugins: inputs, filters and outputs covering the sources and sinks most pipelines need.
• Self-contained gems: every plugin is a Ruby gem published to RubyGems and kept in its own repository under the logstash-plugins organization.
• Writing your own: the contributing guide covers developing and testing new plugins, which the project describes as straightforward.
• Separate issue tracking: plugin bugs and pull requests belong in that plugin's repository, while core work stays in this one.
• Packaged builds: officially released binaries plus Debian and RPM packages for the supported platforms.

Building from source needs JDK 21 with JAVA_HOME set, JRuby, rake and bundler, after which gradle tasks install the development and default gems. The repository holds both the Elastic-licensed X-Pack code and the OSS-licensed code, and an environment variable selects a build made only from the latter.