PART-1 – Setting Up A Home-Lab Elastic ECK

There are wrong spaces later in some of the config-snippets - I have to adjust the pages. Be aware if you just copy and paste stuff from here - some markdown-conversion is not handled well 😐 Setting Up A Home-Lab Elastic ECK I used again Fedora Server (33) to setup a two node Kubernetes cluster... » weiterlesen

PART-2 – ECK setup

PART-2 - ECK setup ECK is only available with basic and enterprise/trial license! Deploy ECK Just run following command to install the custom resource definitions - or download the yaml and apply it kubectl apply -f Setting Up The Elasticsearch-Cluster Create a folder where you save your yaml-files. In this example we will assume... » weiterlesen

PART-6 – Logstash

PART-6 - Logstash Logstash is nice for analyzing the events. Prepare Logstash We will create two different Logstash deployments, so we can scale them up or down individually. One deployment will be for the incoming events, which will simple be forwarded into the RabbitMQ without much groking or so. The second deployment will be mainly... » weiterlesen

PART-5 – Elasticsearch Templates And Policies

PART-5 - Elasticsearch Templates And Policies Because we use Logstash for analysing Logs, we have to prepare templates and ILM policies. Setting Up ILM-Policies And Index-Templates For this setup I created an index-template listening for index-pattern logstash_syslog_ilm-. The field mapping is configured for using the ECS-mapping so you can e.g. simultanously query filebeat, logstash,... » weiterlesen

PART-4 – Prepare RabbitMQ For Buffering Events

PART-4 - Prepare RabbitMQ For Buffering Events The first thought maybe like: What? Why the hell RabbitMQ and not use the persistent queue featuere of Logstash? Because: Input plugins that do not use a request-response protocol cannot be protected from data loss. For example: tcp, udp, zeromq push+pull, and many other inputs do not have... » weiterlesen

PART 3 – Monitoring

PART-3 - Monitoring As we want to know how our ECK is performing, we also want to monitor it with the built in and supplied "Stack Monitoring" Prepare Filebeat For Monitoring The Cluster For this setup you can orient at the - As we have our "ELK" in its own namespace and also a... » weiterlesen

PART-7 – Miscallaneous, Ideas And ToDos

PART-7 - Miscallaneous, Ideas And ToDos A few things to help or make things easier. Upload The YAML-Files e.g. to github or provide a zip-file here (this will take some time - too lazy after finishing this HowTo atm 😉 Usefull Commands See What's Going on kubectl get es,kb,deployments,sts,pods,svc,pv,ingress -o wide -A Execute it with... » weiterlesen