We use Elasticsearch & Kibana for:
- Soluvas Geo. Currently this is the only live production usage. However, in the future, we plan to use Elasticsearch for:
- Centralized logging.
- Performance metrics.
- Operational metrics / time series data.
- Faceted search, full text search, and geospatial search, shadowing the MongoDB data.
- If using Lightsail, it’s required to “Enable Lightsail VPC Peering” so other resources (like ALB) can access it.
Pricing compared to Amazon Elasticsearch:
- Lightsail 1 GB is $5/mo with 40 GB storage
- Amazon Elasticsearch t2.micro Singapore is $20.60/mo ($20.44 + $0.60 1 GB storage) (Note: t2.micro/t3.small does not support reserved instances)
- Amazon Elasticsearch t3.medium Singapore is $81.92/mo with 1 GB storage
- Amazon Elasticsearch t3.medium Singapore Reserved Instance with No Upfront is $65.86/mo with 1 GB storage. Important: Credits cannot apply to Full & Partial Upfront payments!
- Fargate Spot Singapore: vCPU $0.015168/vCPU/hr, RAM $0.001659/GB/hr. 0.25 vCPU ($2.82) + 1 GB ($1.23) = $4.05.
Installing Single-node Elasticsearch & Kibana on Ubuntu VM
The goal is to have:
- Authentication of “
elastic” user be set up with strong password
- Port 9200 serving Elasticsearch HTTP internally inside VPC, connectable from ALB’s target group
- Port 5601 serving Kibana HTTP privately inside VPC, connectable from ALB’s target group
- Port 9243:
/serving Elastic HTTPS both privately and publicly via ALB
/app/kibanaserving Kibana HTTPS both privately and publicly via ALB
- Port 9300 (Java transport client) disabled
Note about TLS: Elasticsearch will not start in multi-node production mode if TLS is disabled. So offloading TLS to ALB is just temporary.
Launch Lightsail VM & Prepare Ubuntu
Required RAM is 1 GB for deb packages. However, if using docker-compose method, required RAM becomes 2 GB, because pure 1 GB will often cause hang when starting docker-compose.
Current configuration as of May 22, 2021:
- elasticsearch and kibana v7.x installed using deb packages.
- Make sure bootstrap.memory_lock is ‘false’.
/etc/default/elasticsearch, control es01 JVM options to -Xms256m -Xmx256m. When using Docker, I have tried 256m and 300m and it did too much GC. Seems to be “more stable” with deb packages instead of Docker.
- Kibana is a node app so there’s no JVM options
Install Elasticsearch & Kibana using Ubuntu Package Repository
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - sudo apt -y install apt-transport-https echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list sudo apt-get update && sudo apt-get -y install elasticsearch # Start Elasticsearch automatically sudo /bin/systemctl daemon-reload sudo /bin/systemctl enable elasticsearch.service sudo systemctl restart elasticsearch.service # Tail journal sudo journalctl -f --unit elasticsearch # Test curl http://localhost:9200/
Important configuration files
- /etc/elasticsearch/elasticsearch.yml. What to edit:
# WARNING: mlockall might cause the JVM or shell session to exit if it tries to allocate more memory than is available!
# bootstrap.memory_lock: ‘true’
- /etc/default/elasticsearch. What to edit:
- /var/lib/elasticsearch (this should be contents of data01/ folder, owned by elasticsearch:elasticsearch)
Configuring memory usage
Edit /etc/default/elasticsearch -> ES_JAVA_OPTS=”-Xms256m -Xmx256″
sudo systemctl restart elasticsearch.service
# This should have been done before, when installing elasticsearch wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - sudo apt-get -y install apt-transport-https echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list sudo apt-get update # Just install kibana sudo apt-get -y install kibana
server.name: es03-sg.lovia.life server.host: 0.0.0.0 server.basePath: /app/kibana server.rewriteBasePath: true server.publicBaseUrl: https://es03-sg.lovia.life:9243/app/kibana elasticsearch.hosts: ['http://es01:9200'] # for v8.x, will change to kibana_system elasticsearch.username: kibana elasticsearch.password: ************** monitoring.cluster_alerts.email_notifications.email_address: [email protected] # Useful for diagnostics # logging.verbose: true
# Kibana to start automatically sudo /bin/systemctl daemon-reload sudo /bin/systemctl enable kibana.service # Restart Kibana sudo systemctl restart kibana.service # Tail journal (not all logs are here) sudo journalctl -f -u elasticsearch -u kibana # Tail kibana.log (detailed logs here) sudo tail -F /var/log/kibana/kibana.log # Test localhost binding curl -v http://localhost:5601/app/kibana/ # Test external IP binding curl -v http://IP_ADDRESS:5601/app/kibana/
Application Load Balancer (ALB) Configuration
For elasticsearch service’s health check:
- Path: /
- Successful status code is 200,401.
For kibana service’s health check:
- Path: /app/kibana/login
Problem: Authentication of [kibana_system] was terminated by realm [reserved] – failed to authenticate user [kibana_system]
Diagnosis: Make sure Kibana can authenticate.
Solution: Do not use kibana_system user before v8.0.
Upgrading Elasticsearch & Kibana
For minor versions, you can use usual apt update and apt upgrade.
For major versions, check official upgrade docs.
Backup & Restore Elasticsearch Cluster (deb packages)
/var/lib/elasticsearch/ folder. (or the entire Lightsail instance)
For configuration folders:
Regenerate / Start A New Basic License
If you get this error/warning/notice in Elasticsearch logs:
blocking [indices:monitor/stats] operation due to expired license. Cluster health, cluster stats and indices stats \noperations are blocked on license expiration. All data operations (read and write) continue to work. \nIf you have a new license, please update it. Otherwise, please reach out to your support contact.
Make sure in Docker Compose YML, license is
basic and not
TimV in this Forum thread:
# HTTP: # curl -uelastic -XPOST 'http://localhost:9200/_xpack/license/start_basic' # We use HTTPS curl -uelastic -XPOST 'https://es01-sg.lovia.life:9243/_xpack/license/start_basic?acknowledge=true'
Automatic S3 Backups
Installing Elasticsearch & Kibana using Docker (did not work with < 2 GB RAM)
It seems that 1 GB struggles when using Docker because of Docker’s overhead, but 2 GB is OK. But 1 GB with Debian package seems to work OK.
Tried to use swapfile but even just 512 MB caused high CPU:
sudo fallocate -l 512MB /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile sudo nano /etc/fstab /swapfile swap swap defaults 0 0 # Check sudo swapon --show
Previously I tried to run with 1 GB by using Docker memory limits but this caused high CPU and hanging:
- Ensuring JVM options is +XX:UseContainerSupport
- Setting es01’s mem_limit to 768m
- Setting kib01’s mem_limit to 384m
After launching the VM, to prepare Ubuntu, see Ubuntu VM.
Docker Host Kernel Configuration
Why is this necessary? Docker production pre-requisites.
sudo nano /etc/sysctl.d/60-vm.conf vm.max_map_count=262144 # Ctrl+X to exit nano sudo service procps start sudo sysctl -w vm.max_map_count=262144
sudo apt -y install apt-transport-https ca-certificates curl gnupg lsb-release curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg echo \ "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt update sudo apt -y install docker-ce docker-ce-cli containerd.io # add ubuntu user to docker group sudo adduser ubuntu docker
Now log out and log in again so docker group permissions take effect.
Install Docker Compose
The recommended way is indeed to install & upgrade as Docker Compose documentation, not from Ubuntu official repositories.
# change the version to latest sudo curl -L https://github.com/docker/compose/releases/download/1.29.2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose docker-compose --version
Troubleshooting (Docker Compose approach)
exited with code 137. This is due to out of memory error.
Make sure docker-compose.yml does not use bootstrap.memory_lock: ‘true’.
Make sure JVM options specifies explicit -Xms and -Xmx that is low compared to available RAM, but sufficient for each container.
Check Docker RAM usage:
docker stats --all
Upgrading Elasticsearch & Kibana on Docker
VERSIONdirectly to latest minor version, with latest patch version
- Docker Compose up:
docker-compose -f elastic-docker-tls.yml up -d
- Check logs
docker-compose -f elastic-docker-tls.yml logs -f
Backup & Restore Elasticsearch Cluster using Docker
As we use Docker Compose, we can simply backup the
data01/ folder. (or the entire Lightsail instance)