Why? What is that good for? Save license-costs and putting ingest-pipeline load somewhere else! :smiley: At my workplace we currently parse the events mostly with Logstash, but configuring those grok-filters so that the messages are ECS-comform is a lot of work. Yes, I know that there is a possibility to export those ingest-pipelines from Beats... » weiterlesen
PART-5 – Elasticsearch Templates And Policies I have to update this page – as the templating stuff is not working as intended for none-legacy indices! Because we use Logstash for analysing Logs, we have to prepare templates and ILM policies. Setting Up ILM-Policies And Index-Templates For this setup I created an index-template listening for index-pattern... » weiterlesen
PART-6 – Logstash Logstash is nice for analyzing the events. Prepare Logstash We will create two different Logstash deployments, so we can scale them up or down individually. One deployment will be for the incoming events, which will simple be forwarded into the RabbitMQ without much groking or so. The second deployment will be mainly... » weiterlesen
PART-2 – ECK setup ECK is only available with basic and enterprise/trial license! Deploy ECK Just run following command to install the custom resource definitions – or download the yaml and apply it kubectl apply -f https://download.elastic.co/downloads/eck/1.3.0/all-in-one.yaml Setting Up The Elasticsearch-Cluster Create a folder where you save your yaml-files. In this example we will assume... » weiterlesen
There are wrong spaces later in some of the config-snippets – I have to adjust the pages. Be aware if you just copy and paste stuff from here – some markdown-conversion is not handled well 😐 Setting Up A Home-Lab Elastic ECK I used again Fedora Server (33) to setup a two node Kubernetes cluster... » weiterlesen
PART-7 – Miscallaneous, Ideas And ToDos A few things to help or make things easier. Upload The YAML-Files e.g. to github or provide a zip-file here (this will take some time – too lazy after finishing this HowTo atm 😉 Usefull Commands See What’s Going on kubectl get es,kb,deployments,sts,pods,svc,pv,ingress -o wide -A Execute it with... » weiterlesen
PART-3 – Monitoring As we want to know how our ECK is performing, we also want to monitor it with the built in and supplied "Stack Monitoring" Prepare Filebeat For Monitoring The Cluster For this setup you can orient at the https://github.com/elastic/cloud-on-k8s/tree/master/config/recipes/beats – As we have our "ELK" in its own namespace and also a... » weiterlesen
PART-4 – Prepare RabbitMQ For Buffering Events You may have a look at Elasticsearch As Log-Buffer The first thought maybe like: What? Why the hell RabbitMQ and not use the persistent queue featuere of Logstash? Because: Input plugins that do not use a request-response protocol cannot be protected from data loss. For example: tcp, udp,... » weiterlesen
