DataDog

It is a SaaS-Based infrastructure monitoring that provides Metrics, Visualization, and Alerting. DataDog charges for the ingestion & retention of logs providing monitoring of servers, databases, tools, and services.

Datadog is great for businesses that need Cloud monitoring, Server performance monitoring service, Server monitoring tools, Server usage analytics, 80+ turn-key integrations for data aggregation

Alert notifications via e-mail and PagerDuty, Full API access, Overlay metrics and events across disparate sources, An easy way to compute rates, ratios, averages, or integrals, Sampling intervals of 10 seconds, Tools for team collaboration

ELK [Elasticsearch Logstash Kibana]

ELK stack allows you to store, classify & visualize your logs in one place. It's an open source for log aggregation purposes consisting of 4 components ElasticSearch, Logstash, Kibana, Beats. It gives you the ability to collect or gather logs from all your systems and applications, analyze these logs, and create visualizations for monitoring, troubleshooting, security analytics, and more.

ElasticSearch Search engine tool where all the logs are stored, indexed and queried

Logstash Collects, logs, and transforms so that it can be easily injected into Elasticsearch

Kibana Visualization tool for your Elasticsearch data, with feature to build your dashboard, filter, navigate

Beats These are Lightweight data shippers, great for data gathering and they sit on your servers as sidecar containers or deploy as functions. Beats gather logs and metrics, and documents them with metadata before shipping them to Logstash or Elasticsearch. There are various types of Beats- Filebeat, MetricBeat, AuditBeat, Heartbeat and others

Based on customer assessments for Log Ingestion below solution is proposed.

Architecture Overview – Logs Ingestion

Steps

• Elasticsearch provides pre-baked images of an elastic agent or filebeat

  • Two types of images are there

    • Normal (enable with monitoring)

    • OSS (Development purpose) you can customize these images based on your need

• Define & register your ELK indexes for your application and configure it in YAML or Terraform templates

• CI-CD pipeline add stage in Jenkinsfile for deploying

  • Elastic Agent or Filebeat in Kubernetes using YAML template

  • EC2 instances using Terraform template

• Integrate these YAML templates in Jenkins deployments job for microservices application in K8s

• Configure Elastic or Filebeat agents as sidecar containers or config map in a microservices application

• Terraform templates in CI-CD pipelines for installation of an elastic agent in EC2 instances

• To scale up this elastic or file beat agent ensure they are part of the autoscaling group or set the desired replica-set

Elastic Agents Single Agent for logs, metrics, security data & threat prevention. It can be deployed in 2 modes

1. Fleet app in Kibana

2. Standalone mode (Install agent, Elastic Agent Docker Image, Elastic Agent in a container)

Log Shippers

• Filebeat

• Elastic Agents

Below are reference templates I have used to implement the ELK setup.

Terraform Templates

Installing Elastic & FileBeat Agents on EC2 instances using terraform module template

Docker Images

Elastic Agent Docker images to deploy in AKS

Filebeat config map as a sidecar on Kubernetes applications

Lambda Function

S3 event triggers the lambda function. Code will get executed whenever a logfile arrives in the S3 bucket

Challenges

• How can you upgrade Elasticsearch as it consists of multiple components & requires re-indexing

• Change in indexes results in re-configuration & installation of underlying Beats, Elastic-agents

• Change the format of a data type as it is Java-based (ex float to date)

• Datastore and restoring data as it will need a lot of time – performance impacts

• Rollback takes a longer time as multiple components are involved

Conclusion

As the visualizer and log analyzer in the ELK system, Kibana includes many functions which are highly compatible with Elasticsearch to help us inject new meaning into raw data. This article just introduces you guys to some basic operations; therefore, you could explore other rich functions in Kibana continuously(There are multiple functions in the newest version of Kibana).

This article just shares a few of Kibana’s usual features with you. If there is something wrong or defects, please leave a message to me.

What’s more, I would mostly welcome your discussion with me on topics of ELK.