Data Processing with Logstash (and Filebeat) – Juhan Aasaru

Salepage link: At HERE. Archive:

What you’ll learn

• Understand the fundamental concepts of Logstash
• Build pipelines that process and manipulates thousands of events
• Send data to Logstash from numerous sources and to several destinations
• Build a fully functional pipeline that handles Apache web server logs
• Process Filebeat events with Logstash

Course content

• Getting Started
• Introduction to the course 
• Introduction to Logstash 
• Installing Logstash on Mac/Linux 
• Installing Logstash on Windows 

Basics of Logstash

• Processing our first event 
• Handling JSON input 
• Outputting events to file 
• Working with HTTP input 
• Filtering events 
• Common filter options 
• Understanding the Logstash execution model 
• Section wrap up 

Project Apache

• Introduction to this section 
• Automatic config reload & file input 
• Parsing requests with Grok 
• Finishing the Grok pattern 
• Accessing field values 
• Formatting dates 
• Setting the time of the event 
• Introduction to conditional statements 
• Working with conditional statements 
• Geographical data enrichment 
• Parsing user agents 
• Finishing up the pipeline 
• Sending processed events to Elasticsearch 
• Handling multiline events 
• Handling multiline events – the easy way 
• Parsing stack traces with Grok 
• Working with @metadata 
• Running multiple pipelines 

Collecting Logs with Filebeat

• Introduction to Beats 
• What we will build 
• Installing Filebeat 
• Configuring Filebeat to use Logstash 
• Enabling the apache module 
• Preparing a simple Logstash pipeline 
• Starting up Filebeat and processing logs 
• Adding the Elasticsearch index template 
• Adding Kibana dashboards 
• Finishing up the pipeline 
• How Filebeat works 
• Clearing the registry 
• Processing more access logs 
• Manual input configuration 
• Evaluation of modules 
• Tagging events 
• Approaches for handling multiple log types 
• Processing Apache error logs 
• Handling multiline logs – approach #1 
• More multiline options 
• Handling multiline logs – approach #2 
• Wrap up 

Conclusion

• You have reached the end!

Requirements

• Familiarity with basic terminal/command prompt usage is a plus

Description

NEW! This course now also includes Filebeat and how to integrate it with Logstash, Elasticsearch, and Kibana!

Want to learn how to process events with Logstash? Then you have come to the right place; this course is by far the most comprehensive course on Logstash here at Udemy! This course specifically covers Logstash, meaning than we can go into much more detail than if this course covered the entire Elastic Stack. So if you want to learn Logstash specifically, then this course is for you!

This course assumes no prior knowledge of or experience with Logstash. We start from the very basics and gradually transition into more advanced topics. The course is designed so that you can follow along the whole time step by step, and you can find all of the configuration files within a GitHub repository. The course covers topics such as handling Apache web server logs (both access and error logs), data enrichment, sending data to Elasticsearch, visualizing data with Kibana, along with covering a number of popular use cases that you are likely to come across. Upon completing this course, you will know all of the most important aspects of Logstash, and will be able to build complex pipeline configurations and process many different kinds of events and data.

What is Logstash?

In case you don’t know what Logstash is all about, it is an event processing engine developed by the company behind Elasticsearch, Kibana, and more. Logstash is often used as a key part of the ELK stack or Elastic Stack, so it offers a strong synergy with these technologies. You can use Logstash for processing many different kinds of events, and an event can be many things. You can process access or error logs from a web server, or you can send events to Logstash from an e-commerce application, such as when an order was received or a payment was processed. You can ingest data from files (flat files, JSON, XML, CSV, etc.), receive data over HTTP or TCP, retrieve data from databases, and more. Logstash then enables you to process and manipulate the events before sending them to a destination of your choice, such as Elasticsearch, e-mail, or Slack.

Why do we need Logstash?

Because by sending events to Logstash, you decouple things. You effectively move event processing out of the web application and into Logstash, meaning that if you need to change how events are processed, you don’t need to deploy a new version of a web application, for instance. The event processing and its configuration is centralized within Logstash instead of every place you trigger events. This means that all the web application needs to do, is to send an event to Logstash; it doesn’t need to know anything about what happens to the event afterwards and where it ends up. This improves your architecture and lets Logstash do what it does best; process events.

Let’s get started!
I hope that you are ready to begin learning Logstash. Have a look around the curriculum if you want to check out the course content in more details. I look forward to seeing you inside the course!

Who this course is for:

• Developers who want to learn Logstash