Skip to main content

Setup Cluster Alerts

© 2024 Elastio Software, Inc. All Rights Reserved.

Setup Cluster Alerts

Elastio
Updated

Contents

Configure Grafana SMTP settings

Within the Elastio command line, perform the following commands to configure email.

ec config grafana
##################### Grafana SMTP Configuration ##################### 
[smtp] 
enabled = false 
host = smtp.domain.com:587 ; SMTP server address and port 
user = email@domain.com ; SMTP username 
password = password ; SMTP password 

skip_verify = true ; Set to true if using self-signed certificates 
from_address = grafana@domain.com ; Sender's email address 
from_name = Elastio On-Prem Cluster ; Sender's name 

[emails] 
welcome_email_on_sign_up = false ; Send welcome email on sign-up ######################################################################

Start Grafana

http://leader_node_ip_or_hostname:3001
Default credentials admin/admin, on first login you must change the password

Setup Alerts in Grafana

Navigate to Alerting> Contact points. Press the “Add contact point” button.

After selecting integration “Email” and specifying addresses, you can press the “Test” button to send a test notification to the contact point.

Note: The alerts will be set with default thresholds. For example, Elastio will trigger an alert if free disk space is less than 10 GB by default. You can modify the thresholds for alerts based on your needs.

Modify Alert Thresholds

To modify the default thresholds for these alerts, follow these steps:

Navigate to Alerting → Alert rules, and duplicate an alert from Elastio Provisioned Rules 

  • Set the thresholds that suit your needs.
  • Suppress the old alert created by default.
View Alert Status

Alert statuses can be seen on the Elastio Cluster Metrics dashboard in Grafana.

Enabling HTTPS in Grafana

The setup relies on detecting TLS certificate and key files inside the Grafana data directory; if they exist, Grafana will automatically switch to HTTPS when starting.

Prerequisites

Resolvable Hostname
Ensure the machine (master node) has a resolvable hostname (e.g., grafana.example.com in the following examples).

Steps to Enable HTTPS

  • Create a tls directory
    By default, there is no tls folder. Create one inside the host’s Grafana data directory and navigate into it:
sudo -i
mkdir -p /var/lib/elastio/grafana/data/tls
cd /var/lib/elastio/grafana/data/tls
  • Obtain a Certificate and Key
    You normally want to use a certificate from a trusted CA. However, for testing or internal use, you can also create a self-signed certificate.

Example: Generating a Self-Signed Certificate

openssl req -newkey rsa:2048 -nodes -keyout grafana.key \
    -x509 -days 365 -out grafana.crt \
    -subj "/CN=grafana.example.com"

You need to store two files in that directory:

  1. grafana.key (private key)
  2. grafana.crt (certificate)
  • Set File Permissions
    To make these files readable by Grafana, you need to change their permissions:
chmod 644 grafana.crt grafana.key

Or, to make them readable only by the Grafana user inside the container:

chown 472 grafana.crt grafana.key
chmod 640 grafana.crt grafana.key
  • Restart Grafana
docker service update --force elastio_grafana

Usage

Grafana will now work on HTTPS:

https://<your-host-or-ip>:3001

Related articles