Welcome!

@CloudExpo Authors: Andy Thurai, Yeshim Deniz, Ed Featherston, Pat Romanski, Liz McMillan

Related Topics: @DevOpsSummit, Linux Containers, Containers Expo Blog

@DevOpsSummit: Article

Five #Logstash Alternatives | @DevOpsSummit @Sematext #Elasticsearch

Shippers have their pros and cons, and ultimately it’s down to your specifications

When it comes to centralizing logs to Elasticsearch, the first log shipper that comes to mind is Logstash. People hear about it even if it's not clear what it does:
- Bob: I'm looking to aggregate logs
- Alice: you mean... like... Logstash?

When you get into it, you realize centralizing logs often implies a bunch of things, and Logstash isn't the only log shipper that fits the bill:

  • fetching data from a source: a file, a UNIX socket, TCP, UDP...
  • processing it: appending a timestamp, parsing unstructured data, adding Geo information based on IP
  • shipping it to a destination. In this case, Elasticsearch. And because Elasticsearch can be down or struggling, or the network can be down, the shipper would ideally be able to buffer and retry

In this post, we'll describe Logstash and its alternatives - 5 "alternative" log shippers (Filebeat, Fluentd, rsyslog, syslog-ng and Logagent), so you know which fits which use-case.

Logstash
It's not the oldest shipper of this list (that would be syslog-ng, ironically the only one with "new" in its name), it's certainly the best known. That's because it has lots of plugins: inputs, codecs, filters and outputs. Basically, you can take pretty much any kind of data, enrich it as you wish, then push it to lots of destinations.

Strengths
Logstash's main strongpoint is flexibility, due to the number of plugins. Also, its clear documentation and straightforward configuration format means it's used in a variety of use-cases. This leads to a virtuous cycle: you can find online recipes for doing pretty much anything. Here are a few examples from us: 5 minute intro, reindexing data in Elasticsearch, parsing Elasticsearch logs, rewriting Elasticsearch slowlogs so you can replay them with JMeter.

Weaknesses
Logstash's Achille's heel has always been performance and resource consumption (the default heap size is 1GB). Though performance improved a lot over the years, it's still a lot slower than the alternatives. We've done some benchmarks comparing Logstash to rsyslog and to filebeat and Elasticsearch's Ingest node. This can be a problem for high traffic deployments, when Logstash servers would need to be comparable with the Elasticsearch ones.

Another problem is that Logstash currently doesn't buffer yet. A typical workaround is to use Redis or Kafka as a central buffer:

Logstash - Kafka - Elasticsearch

Typical use-case
Because of the flexibility and abundance of recipes, Logstash is a great tool for prototyping, especially for more complex parsing. If you have big servers, you might as well install Logstash on each. You won't need buffering if you're tailing files, because the file itself can act as a buffer (i.e. Logstash remembers where it left off):

Logstash - Elasticsearch (1)

If you have small servers, installing Logstash on each is a no go, so you'll need a lightweight log shipper on them, that could push data to Elasticsearch though one (or more) central Logstash servers:

Light shipper - Logstash - Elasticsearch

As your logging project moves forward, you may or may not need to change your log shipper because of performance/cost. When choosing whether Logstash performs well enough, it's important to have a good estimation of throughput needs - which would predict how much you'd spend on Logstash hardware.

Filebeat
As part of the Beats "family", Filebeat is a lightweight log shipper that came to life precisely to address the weakness of Logstash: Filebeat was made to be that lightweight log shipper that pushes to Logstash.

With version 5.x, Elasticsearch has some parsing capabilities (like Logstash's filters) called Ingest. This means you can push directly from Filebeat to Elasticsearch, and have Elasticsearch do both parsing and storing. You shouldn't need a buffer when tailing files because, just as Logstash, Filebeat remembers where it left off:

Filebeat - Ingest - Elasticsearch

If you need buffering (e.g. because you don't want to fill up the file system on logging servers), you can use Redis/Kafka, because Filebeat can talk to them:

Filebeat - Kafka - Elasticsearch

Strengths
Filebeat is just a tiny binary with no dependencies. It takes very little resources and, though it's young, I find it quite reliable - mainly because it's simple and there are few things that can go wrong. That said, you have lots of knobs regarding what it can do. For example, how aggressive it should be in searching for new files to tail and when to close file handles when a file didn't get changes for a while.

Weaknesses
Filebeat's scope is very limited, so you'll have a problem to solve somewhere else. For example, if you use Logstash down the pipeline, you have about the same performance issue. Because of this, Filebeat's scope is growing. Initially it could only send logs to Logstash and Elasticsearch, but now it can send to Kafka and Redis, and in 5.x it also gains filtering capabilities.

Typical use-cases
Filebeat is great for solving a specific problem: you log to files, and you want to either:

  • ship directly to Elasticsearch. This works if you want to just "grep" them or if you log in JSON (Filebeat can parse JSON). Or, if you want to use Elasticsearch's Ingest for parsing and enriching (assuming the performance and functionality of Ingest fits your needs)
  • put them in Kafka/Redis, so another shipper (e.g. Logstash, or a custom Kafka consumer) can do the enriching and shipping. This assumes that the chosen shipper fits your functionality and performance needs

Logagent
This is our log shipper that was born out of the need to make it easy for someone who didn't use a log shipper before to send logs to Logsene (our logging SaaS which exposes the Elasticsearch API). And because Logsene exposes the Elasticsearch API, Logagent can be just as easily used to push data to Elasticsearch.

Strengths
The main one is ease of use: if Logstash is easy (actually, you still need a bit of learning if you never used it, that's natural), this one really gets you started in a minute. It tails everything in /var/log out of the box, parses various logging formats out of the box (Elasticsearch, Solr, MongoDB, Apache HTTPD...). It can mask sensitive data like PII, date of birth, credit card numbers, etc. It will also do GeoIP enriching based on IPs (e.g., for access logs) and update the GeoIP database automatically. It's also light and fast, you'll be able to put it on most logging boxes (unless you have very small ones, like appliances). The new 2.x version added support for pluggable inputs and outputs in a form of 3rd party node.js modules. Very importantly, Logagent has local buffering so, unlike Logstash, it will not lose your logs when the destination is not available.

Weaknesses
Logagent is still young, although is developing and maturing quickly. It has some interesting functionality (e.g. it accepts Heroku or CloudFoundry logs), but it is not yet as flexible as Logstash.

Typical use-cases
Logagent is a good choice of a shipper that can do everything (tail, parse, buffer - yes, it can buffer on disk - and ship) that you can install on each logging server. Especially if you want to get started quickly. Logagent is embedded in Sematext Docker Agent to parse and ship Docker containers logs. Sematext Docker Agent works with Docker Swarm, Docker Datacenter, Docker Cloud, as well as Amazon EC2, Google Container Engine, Kubernetes, Mesos, RancherOS, and CoreOS, so for Docker log shipping, this is the tool to use.

rsyslog
The default syslog daemon on most Linux distros, rsyslog can do so much more than just picking logs from the syslog socket and writing to /var/log/messages. It can tail files, parse them, buffer (on disk and in memory) and ship to a number of destinations, including Elasticsearch. You can find a howto for processing Apache and system logs here.

Strengths
rsyslog is the fastest shipper that we tested so far. If you use it as a simple router/shipper, any decent machine will be limited by network bandwidth, but it really shines when you want to parse multiple rules. Its grammar-based parsing module (mmnormalize) works at constant speed no matter the number of rules (we tested this claim). This means that with 20-30 rules, like you have when parsing Cisco logs, it can outperform the regex-based parsers like grok by a factor of 100 (it can be more or less, depending on the grok implementation and liblognorm version).

It's also one of the lightest parsers you can find, depending on the configured memory buffers.

Weaknesses
rsyslog requires more work to get the configuration right (you can find some sample configuration snippets here on our blog) and this is made more difficult by two things:

  • documentation is hard to navigate, especially for somebody new to the terminology
  • versions up to 5.x had a different configuration format (expanded from the syslogd config format, which it still supports). Newer versions can still work with the old format, but most newer features (like the Elasticsearch output) only work with the new configuration format, but then again there are older plugins (for example, the Postgres output) which only support the old format

Though rsyslog tends to be reliable once you get to a stable configuration (and it's rich enough that there are usually multiple ways of getting the same result), you're likely to find some interesting bugs along the way. Not all features are tested as part of the testbench.

Typical use-cases
rsyslog fits well in scenarios where you either need something very light yet capable (an appliance, a small VM, collecting syslog from within a Docker container). If you need to do processing in another shipper (e.g. Logstash) you can forward JSON over TCP for example, or connect them via a Kafka/Redis buffer.

rsyslog also works well when you need that ultimate performance. Especially if you have multiple parsing rules. Then it makes sense to invest time in getting that configuration working.

syslog-ng
You can think of syslog-ng as an alternative to rsyslog (though historically it was actually the other way around). It's also a modular syslog daemon, that can do much more than just syslog. It recently received disk buffers and an Elasticsearch HTTP output. Equipped with a grammar-based parser (PatternDB), it has all you probably need to be a good log shipper to Elasticsearch.

Advantages
Like rsyslog, it's a light log shipper and it also performs well. It used to be a lot slower than rsyslog before, and I haven't benchmarked the two recently, but 570K logs/s two years ago isn't bad at all. Unlike rsyslog, it features a clear, consistent configuration format and has nice documentation.

Disadvantages
The main reason why distros switched to rsyslog was syslog-ng Premium Edition, which used to be much more feature-rich than the Open Source Edition which was somewhat restricted back then. We're concentrating on the Open Source Edition here, all these log shippers are open source. Things have changed in the meantime, for example disk buffers, which used to be a PE feature, landed in OSE. Still, some features, like the reliable delivery protocol (with application-level acknowledgements) have not made it to OSE yet.

Typical use-cases
Similarly to rsyslog, you'd probably want to deploy syslog-ng on boxes where resources are tight, yet you do want to perform potentially complex processing. As with rsyslog, there's a Kafka output that allows you to use Kafka as a central queue and potentially do more processing in Logstash or a custom consumer:

syslog-ng - Kafka - Elasticsearch

The difference is, syslog-ng has an easier, more polished feel than rsyslog, but likely not that ultimate performance: for example, only outputs are buffered, so processing is done before buffering - meaning that a processing spike would put pressure up the logging stream.

Fluentd
Fluentd was built on the idea of logging in JSON wherever possible (which is a practice we totally agree with) so that log shippers down the line don't have to guess which substring is which field of which type. As a result, there are libraries for virtually every language, meaning you can easily plug in your custom applications to your logging pipeline.

Advantages
Like most Logstash plugins, Fluentd plugins are in Ruby and very easy to write. So there are lots of them, pretty much any source and destination has a plugin (with varying degrees of maturity, of course). This, coupled with the "fluent libraries" means you can easily hook almost anything to anything using Fluentd.

Disadvantages
Because in most cases you'll get structured data through Fluentd, it's not made to have the flexibility of other shippers on this list (Filebeat excluded). You can still parse unstructured via regular expressions and filter them using tags, for example, but you don't get features such as local variables or full-blown conditionals. Also, while performance is fine for most use-cases, it's not in on the top of this list: buffers exist only for outputs (like in syslog-ng), single-threaded core and the Ruby GIL for plugins means ultimate performance on big boxes is limited, but resource consumption is acceptable for most use-cases. For small/embedded devices, you might want to look at Fluent Bit, which is to Fluentd similar to how Filebeat is for Logstash.

Typical use-cases
Fluentd is a good fit when you have diverse or exotic sources and destinations for your logs, because of the number of plugins. Also, if most of the sources are custom applications, you may find it easier to work with fluent libraries than coupling a logging library with a log shipper. Especially if your applications are written in multiple languages - meaning you'd use multiple logging libraries, which may behave differently.

The conclusion?
First of all, the conclusion is that you're awesome for reading all the way to this point. If you did that, you get the nuances of an "it depends on your use-case" kind of answer. All these shippers have their pros and cons, and ultimately it's down to your specifications (and in practice, also to your personal preferences) to choose the one that works best for you. If you need help deciding, integrating, or really any help with logging don't be afraid to reach out - we offer Logging Consulting. Similarly, if you are looking for a place to ship your logs and avoid costs/headaches associated with running the full ELK/Elastic Stack on your own servers, check out Logsene - it exposes Elasticsearch API, so you can use it with all shippers we covered here.

The post 5 Logstash Alternatives appeared first on Sematext.

More Stories By Radu Gheorghe

Radu Gheorghe is a search consultant, software engineer and trainer at Sematext Group, working mainly with Elasticsearch, Solr and logging-related projects. He is the co-author of Elasticsearch in Action.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@CloudExpo Stories
Andi Mann, Chief Technology Advocate at Splunk, is an accomplished digital business executive with extensive global expertise as a strategist, technologist, innovator, marketer, and communicator. For over 30 years across five continents, he has built success with Fortune 500 corporations, vendors, governments, and as a leading research analyst and consultant.
You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technologies.
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settl...
With tough new regulations coming to Europe on data privacy in May 2018, Calligo will explain why in reality the effect is global and transforms how you consider critical data. EU GDPR fundamentally rewrites the rules for cloud, Big Data and IoT. In his session at 21st Cloud Expo, Adam Ryan, Vice President and General Manager EMEA at Calligo, examined the regulations and provided insight on how it affects technology, challenges the established rules and will usher in new levels of diligence arou...
For organizations that have amassed large sums of software complexity, taking a microservices approach is the first step toward DevOps and continuous improvement / development. Integrating system-level analysis with microservices makes it easier to change and add functionality to applications at any time without the increase of risk. Before you start big transformation projects or a cloud migration, make sure these changes won’t take down your entire organization.
Digital transformation has increased the pace of business creating a productivity divide between the technology haves and have nots. Managing financial information on spreadsheets and piecing together insight from numerous disconnected systems is no longer an option. Rapid market changes and aggressive competition are motivating business leaders to reevaluate legacy technology investments in search of modern technologies to achieve greater agility, reduced costs and organizational efficiencies. ...
Organizations planning enterprise data center consolidation and modernization projects are faced with a challenging, costly reality. Requirements to deploy modern, cloud-native applications simultaneously with traditional client/server applications are almost impossible to achieve with hardware-centric enterprise infrastructure. Compute and network infrastructure are fast moving down a software-defined path, but storage has been a laggard. Until now.
DXWorldEXPO LLC announced today that Kevin Jackson joined the faculty of CloudEXPO's "10-Year Anniversary Event" which will take place on November 11-13, 2018 in New York City. Kevin L. Jackson is a globally recognized cloud computing expert and Founder/Author of the award winning "Cloud Musings" blog. Mr. Jackson has also been recognized as a "Top 100 Cybersecurity Influencer and Brand" by Onalytica (2015), a Huffington Post "Top 100 Cloud Computing Experts on Twitter" (2013) and a "Top 50 C...
In his session at 20th Cloud Expo, Mike Johnston, an infrastructure engineer at Supergiant.io, discussed how to use Kubernetes to set up a SaaS infrastructure for your business. Mike Johnston is an infrastructure engineer at Supergiant.io with over 12 years of experience designing, deploying, and maintaining server and workstation infrastructure at all scales. He has experience with brick and mortar data centers as well as cloud providers like Digital Ocean, Amazon Web Services, and Rackspace. H...
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
Digital Transformation is much more than a buzzword. The radical shift to digital mechanisms for almost every process is evident across all industries and verticals. This is often especially true in financial services, where the legacy environment is many times unable to keep up with the rapidly shifting demands of the consumer. The constant pressure to provide complete, omnichannel delivery of customer-facing solutions to meet both regulatory and customer demands is putting enormous pressure on...
More and more brands have jumped on the IoT bandwagon. We have an excess of wearables – activity trackers, smartwatches, smart glasses and sneakers, and more that track seemingly endless datapoints. However, most consumers have no idea what “IoT” means. Creating more wearables that track data shouldn't be the aim of brands; delivering meaningful, tangible relevance to their users should be. We're in a period in which the IoT pendulum is still swinging. Initially, it swung toward "smart for smart...
"We were founded in 2003 and the way we were founded was about good backup and good disaster recovery for our clients, and for the last 20 years we've been pretty consistent with that," noted Marc Malafronte, Territory Manager at StorageCraft, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Here are the Top 20 Twitter Influencers of the month as determined by the Kcore algorithm, in a range of current topics of interest from #IoT to #DeepLearning. To run a real-time search of a given term in our website and see the current top influencers, click on the topic name. Among the top 20 IoT influencers, ThingsEXPO ranked #14 and CloudEXPO ranked #17.
"DivvyCloud as a company set out to help customers automate solutions to the most common cloud problems," noted Jeremy Snyder, VP of Business Development at DivvyCloud, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
In his session at 21st Cloud Expo, Michael Burley, a Senior Business Development Executive in IT Services at NetApp, described how NetApp designed a three-year program of work to migrate 25PB of a major telco's enterprise data to a new STaaS platform, and then secured a long-term contract to manage and operate the platform. This significant program blended the best of NetApp’s solutions and services capabilities to enable this telco’s successful adoption of private cloud storage and launching o...
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant tha...
As data explodes in quantity, importance and from new sources, the need for managing and protecting data residing across physical, virtual, and cloud environments grow with it. Managing data includes protecting it, indexing and classifying it for true, long-term management, compliance and E-Discovery. Commvault can ensure this with a single pane of glass solution – whether in a private cloud, a Service Provider delivered public cloud or a hybrid cloud environment – across the heterogeneous enter...
The Jevons Paradox suggests that when technological advances increase efficiency of a resource, it results in an overall increase in consumption. Writing on the increased use of coal as a result of technological improvements, 19th-century economist William Stanley Jevons found that these improvements led to the development of new ways to utilize coal. In his session at 19th Cloud Expo, Mark Thiele, Chief Strategy Officer for Apcera, compared the Jevons Paradox to modern-day enterprise IT, examin...
DXWorldEXPO LLC announced today that All in Mobile, a mobile app development company from Poland, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. All In Mobile is a mobile app development company from Poland. Since 2014, they maintain passion for developing mobile applications for enterprises and startups worldwide.
DXWorldEXPO LLC announced today that ICC-USA, a computer systems integrator and server manufacturing company focused on developing products and product appliances, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City. ICC is a computer systems integrator and server manufacturing company focused on developing products and product appliances to meet a wide range of ...