Logz.io: collection logs from Kubernetes — fluentd vs filebeat

We are using Logz.io to collect our Kubernetes cluster logs (also, there is a local Loki instance).

Logs are collected and processed by a Fluentd pod on every WorkerNode which are deployed from a DaemonSet in its default configuration, see the documentation here — logzio-k8s.

The problem we faced is that those pods are consuming too much CPU — up to 3000 millicpu, while our WorkerNodes has only 4 cores, e.g. 4000 millicpu.

So, to solve this issue I’ve decided to search for similar log collectors and the second thing to do is to able to deploy them with Ansible via a Helm chart, but Fluentd has no ready charts yet.

The found solution w a Filebeat collector, see Ship k8s logs with Helm via Filebeat — let’s try it.

At first, we will deploy it manually to a Dev cluster, and then we’ll add an Ansible task to deploy it to our Production.

Logz.io Filebeat Helm chart

Add a repository:

$ helm repo add logzio-helm [https://logzio.github.io/logzio-helm/filebeat](https://logzio.github.io/logzio-helm/filebeat)
“logzio-helm” has been added to your repositories

Next, we need to find our token and region — go to the account’s General Settings:

As we have the default region — us-east-1, see How to look up your account region, then we can remove the secrets.logzioRegion from the Helm chart's arguments.

Also, by default Logz.io client will be installed in the kube-system namespace, but I'd like to place it to a dedicated NS to make it easier to monitor its used resources.

The Namespace is configured in the values file so let’s override it with --set, plus let's add the --create-namespace and --debug options:

$ helm install — namespace=dev-1–18-devops-logzio-ns \
 --create-namespace — debug \
 --set secrets.logzioShippingToken='AVG***Onq' \
 --set secrets.clusterName='bttrm-eks-dev-1–18' \
 --set namespace=dev-1–18-devops-logzio-ns \
logzio-k8s-logs logzio-helm/logzio-k8s-logs

Check pods:

$ kubectl -n dev-1–18-devops-logzio-ns get pod
NAME READY STATUS RESTARTS AGE
filebeat-2qt5s 1/1 Running 0 96s
filebeat-4xb44 1/1 Running 0 96s
filebeat-9prr9 1/1 Running 0 96s
filebeat-cth47 1/1 Running 0 96s
filebeat-fgmgx 1/1 Running 0 96s
filebeat-gb5ts 1/1 Running 0 96s
filebeat-hs9tr 1/1 Running 0 96s
filebeat-nskvg 1/1 Running 0 96s
filebeat-wfgbg 1/1 Running 0 96s

And logs in the Kibana on the app.logs.io page:

Ansible

There is a post about deploying a Helm chart with Ansible — Ansible: модуль community.kubernetes и установка Helm-чарта с ExternalDNS (Rus), at this time just a quick example.

Add a when: "eks_env.startswith('prod')" condition, as we will deploy Logz.io to only the Production cluster:

- name: "Add Logzio chart repo"
  when: "eks_env.startswith('prod')"
  community.kubernetes.helm_repository:
    name: "logzio-helm"
    repo_url: "https://logzio.github.io/logzio-helm/filebeat"

- name: "Deploy Logzio Filebit chart to the {{ eks_env }}-devops-logzio-ns namespace"
  when: "eks_env.startswith('prod')"
  community.kubernetes.helm:
    kubeconfig: "{{ kube_config_path }}"
    name: "logzio-k8s-logs"
    chart_ref: "logzio-helm/logzio-k8s-logs"
    release_namespace: "{{ eks_env }}-devops-logzio-ns"
    create_namespace: true
    values:
      secrets:
        logzioShippingToken: "{{ logzio_token }}"
        clusterName: "{{ eks_cluster_name }}"
      namespace: "{{ eks_env }}-devops-logzio-ns"

Specify the logzio_token in the Ansible variables and encrypt it with the ansible-vault, as we have our Ansible roles stored in our GitHub repository.

Fluentd vs Filebeat — CPU and performance

And finally, CPU usage: old fluentd (Ruby + C) on the left side vs new filebeat (Golang) on the right side:

Originally published at RTFM: Linux, DevOps and system administration.