Elastalert: implementing rich monitoring with Elasticsearch

Standard

Hi, dear readers! Welcome to my blog. On this post, we will take a tour on a open source project developed by Yelp, called Elastalert. Focused on enriching Elasticsearch’s role as a monitoring tool, it allow us to query Elasticsearch, sending alerts to different types of tools, such as e-mail boxes, Telegram chats, JIRA issues and more. So, without further delay, let’s go deep on the tool!

Set up

In order to set up Elastalert, we need to clone the project’s Git repository and install it with Python. If the reader doesn’t have Python or Git installed, I recommend following the instructions here for Python and here for Git. For this tutorial, I am using a Unix OS, but the instructions are similar for other environments such as Linux. Also, on this tutorial I am using virtulenv, in order to keep my Python interpreter “clean”. The reader can find instructions to install virtualenv here.

To display the alerts, we will use a Telegram channel, which will receive alerts sent by a Telegram bot. In order to prepare the bot, we need a Telegram account and use the Bot Father (@BotFather) to create the bot, then create a public channel on telegram and associate the bot on the channel’s admins. The instructions to make this configurations can be found here. In order to easy the steps for the reader, I leave the bot created for this lab (@elastalerthandson) published for anyone who wants to use this bot on his own telegram channels for testing!

With all the tools installed and ready, let’s begin by cloning the Elastalert Git repository. To do this, we run the following command, on the folder of our choice:

git clone https://github.com/Yelp/elastalert.git

After running the command, we will see that a folder called “elastalert” was created. Before we proceed, we will also create a virtualenv environment, where we will install Elastalert. We do this by running:

virtualenv virtualenvelastalert

After creating the virtual environment – which will create a folder called “virtualenvelastalert” -, we need to activate it before we proceed with the install. To do this, we run the following command, assuming the reader is on the same folder of the previous command:

source virtualenvelastalert/bin/activate

After activating, we will notice that the name of our virtual environment is now written as a prefix on the shell, meaning that it is activated. Now, to install elastalert, we navigate to the folder created previously by our git clone command and type the following:

python setup.py install
sudo pip install -r requirements.txt

That’s it! Now that we have Elastalert installed, let’s continue the setup by creating the Elasticsearch index that it will be used as a metadata repository by Elastalert.

Creating the metadata index

In order to create Elastalert’s index, we run the command:

elastalert-create-index

The command-line tool will ask us some settings such as the name we want for the index and the ip/port of our Elasticsearch’s cluster. After providing the settings, the tool will create the index, like we can see on the picture bellow:Creating the configuration files

All the configuration on Elastalert is made by YAML files. The main configuration file for the tool is called by default as “config.yaml” and is located on the same folder where we start Elastalert – which we will do in some moments. For our main configuration file, let’s create a file called “config.yaml” like the following:

rules_folder: rules_folder

run_every:
  seconds: 40

buffer_time:
  minutes: 15

es_host: 192.168.99.100

es_port: 9200

writeback_index: elastalert_status

alert_time_limit:
  days: 2

On the config above, we defined:

  • The rules_folder property which defines the folder where our rules will be (all YAML files on the folder will be processed);
  • The run_every property will make Elastalert to run all the rules on a 40 seconds frequency;
  • The buffer_time property will make Elastalert cache the last period of time defined by the range of the property. This approach is used when the queries made on Elasticsearch are not on real time data;
  • The host ip of the Elasticsearch’s node used to query the alerts;
  • The host port of the Elasticsearch’s node used to query the alerts;
  • The index used to store the metadata, that we created on the previous section;
  • The maximum period of time Elastalert will hold a alert that the delivery has failed, making retries during the period;

Now, let’s create the “rules_folder” folder and create 3 YAML files, which will hold our rules:

  • twitter_flatline.yaml
  • twitter_frequency.yaml
  • twitter-blacklist.yaml

On this rules, we will test 3 types of rules Elastalert can manage:

  • The flatline rule, which will alert when the number of documents find for a search drop bellow a threshold;
  • The frequency rule, which will alert when a number of documents for a certain period of time is reached;
  • The blacklist rule, which will alert when any document containing a list of words is found on the timeframe collected by the tool;

Of course, there’s other rule types alongside those that we will cover on this lab, like the spike rule that can detect abnormal grows or shrinks on data across a time period, or the whitelist rule, which alert on any documents that contain any words from a list. More information about rules and their types can be found at the references on the end of this post.

For this lab, we will use a elasticsearch index with twitter data. The reader can found more information about how to set up a ELK environment on my ELK series. The Logstash configuration file used on this lab is as follows:

input {
      twitter {
        consumer_key => "XXXXXXXXXXXXXXXXX"
        consumer_secret => "XXXXXXXXXXXXXXXX"
        keywords => ["coca cola","java","elasticsearch","amazon"]
        oauth_token => "XXXXXXXXXXXXXXX"
        oauth_token_secret => "XXXXXXXXXXXXXX"
    }
}



 output {
      stdout { codec => rubydebug }
      elasticsearch {
            hosts => [ "192.168.99.100:9200" ]
            index => "twitter-%{+YYYY.MM.dd}"
        }
}

With our ELK stack set up and running, let’s begin creating the rules. First, we create the frequency rule, by configuring the respective YAML file with the following code:

name: Twitter frequency rule

type: frequency

index: twitter-*

num_events: 3

timeframe:
  minutes: 15

realert:
  hours: 2

filter:
- query:
   query_string:
    query: "message:amazon"


alert:
- "telegram"

telegram_bot_token: 184186982:AAGpJRyWQ2Rb_RcFXncGrJrBrSK7BzoVFU8

telegram_room_id: "@elastalerthandson"

On the following file we configure a frequency rule. The rule is configured by setting the following properties:

  • name: This property defines the rule’s name. This property acts as the rule ID;
  • type: This property defines the type of rule we are creating;
  • index: This property defines the index on Elasticsearch where we want to make the searches;
  • num_events: The number of documents necessary to be found in order to fire the alert;
  • timeframe: The time period which will be queried to check the rule;
  • realert: This property defines the time period that Elastalert will stop realerting the rule after the first match, preventing the users to be flooded with alerts;
  • filter: This property is where we configure the query that will be send to Elasticsearch in order to check the rule;
  • alert: This property is a list of targets which we want our alerts to be send. On our case, we just defined the telegram target;
  • telegram_bot_token: On this property we set the access token from our bot, as received by the Botfather;
  • telegram_room_id: On this property we define the id of the channel we want the alerts to be sent;

As we can see, is a very straightforward and simple configuration file. For the flatline config, we configure our respective YAML as follows:

name: Twitter flatline rule

type: flatline

index: twitter-*

threshold: 30

timeframe: 
 minutes: 5

realert:
 minutes: 30

use_count_query: true

doc_type: logs

alert:
- "telegram"

telegram_bot_token: 184186982:AAGpJRyWQ2Rb_RcFXncGrJrBrSK7BzoVFU8

telegram_room_id: "@elastalerthandson"

The configuration is pretty much the same of the previous file, with the exception of 3 new properties:

  • threshold: This property defines the minimum amount of documents expected for the rule to receive in order that a alert is not needed to be sent;
  • use_count_query: This property defines that Elastalert must use the count API from Elasticsearch. This API returns just the number of documents for the rule to be validated, eliminating the need to process the query data;
  • doc_type: This property is needed by the count API aforementioned, in order to query the document count for a specific document type;

Finally, let’s configure our final rule, coding the final YAML as follows:

name: Twitter blacklist rule

type: blacklist

index: twitter-*

compare_key: message

blacklist:
- "android"
- "java"

realert:
  hours: 4

filter:
- query:
   query_string:
    query: "*"

alert:
- "telegram"

telegram_bot_token: 184186982:AAGpJRyWQ2Rb_RcFXncGrJrBrSK7BzoVFU8

telegram_room_id: "@elastalerthandson"

On this file, the new properties that we needed to configure are:

  • compare_key: This property defines the field on the documents that Elastalert will check the blacklist;
  • blacklist: This property is a list of words which Elastalert will compare against the documents in order to check if any document has a blacklisted word;

And that concludes our configuration. Now, let’s run Elastalert!

Running Elastalert

To run Elastalert, all we need to do is run a command like this, on the same folder of our YAML structure – where “config.yaml” is located:

elastalert --start NOW --verbose

On the command above, we set the flag “–start” to define that we want Elastalert to start the measurings from now up and the “–verbose” flag to print all the info log messages.

The simplest of the rules to test it out is the flatline rule. All we have to do is wait for about 5 minutes with Elasticsearch running and Logstash stopped – so no documents are streaming. After the wait, we can see on our channel that a alert is received on the channel:

And, as the time passes, we will receive other alerts as well, like the frequency alert:

Conclusion

And so we conclude our tutorial about Elastalert. With a simple usage, we can see that we can construct really powerful alerts for our Elasticsearch system, enforcing the rule of the search engine on a monitoring ecosystem. Thank you for following me on another post, until next time.

Elastalert Git repository

Elastalert documentation

Lab’s source-code

34 thoughts on “Elastalert: implementing rich monitoring with Elasticsearch”

  2. Prnash

    i tried to install elastalert but got the following error

    blist/_blist.c: In function ‘init_blist_types1’:
    blist/_blist.c:7376: error: expected expression before ‘)’ token
    blist/_blist.c:7378: error: ‘PyBList_Type’ undeclared (first use in this function)
    blist/_blist.c:7378: error: ‘PyType_Type’ undeclared (first use in this function)
    blist/_blist.c:7379: error: ‘PyRootBList_Type’ undeclared (first use in this function)
    blist/_blist.c:7380: error: ‘PyBListIter_Type’ undeclared (first use in this function)
    blist/_blist.c:7381: error: ‘PyBListReverseIter_Type’ undeclared (first use in this function)
    blist/_blist.c: In function ‘init_blist_types2’:
    blist/_blist.c:7394: warning: implicit declaration of function ‘PyType_Ready’
    blist/_blist.c:7394: error: ‘PyRootBList_Type’ undeclared (first use in this function)
    blist/_blist.c:7395: error: ‘PyBList_Type’ undeclared (first use in this function)
    blist/_blist.c:7396: error: ‘PyBListIter_Type’ undeclared (first use in this function)
    blist/_blist.c:7397: error: ‘PyBListReverseIter_Type’ undeclared (first use in this function)
    blist/_blist.c: At top level:
    blist/_blist.c:7404: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘init_blist’
    error: Setup script exited with error: command ‘gcc’ failed with exit status 1

    Like

  3. Hi,

    You mentioned the following: “In order to prepare the bot, we need a Telegram account and use the Bot Father (@BotFather) to create the bot, then create a public channel on telegram and associate the bot on the channel’s admins.”

    However, it is not entirely clear how to do this and what you mean.

    I have created a Telegram account. Using @FatherBot created a another bot and received the API token.

    In the rule yaml I have configured the following:

    telegram_bot_token: tokenwithoutquotes
    telegram_room_id: “@my_telegram_user_id”

    I expected to receive alerts on @my_telegram_user_id

    Is this correct? Or what is wrong?

    Thanks

    Elmar

    Like

    • Hi, Elmar! Elmar, after you create the bot using the @FatherBot, on the telegram_room_id, is not your user, you need to create a channel on telegram, associate your bot to the channel with admin rights, and them use the channel’s ID on the telegram_room_id, this way the alerts will be pushed to the channel, ok?

      Thank you

      Like

  5. Elastauser

    I got the following error while running the python setup.py install

    blist/_blist.c:38:20: fatal error: Python.h: No such file or directory

    I solved it by running the following command

    yum -y install python-devel

    I spent a bit finding so I hope this helps someone.

    Liked by 1 person

  6. meia

    i got error,
    [root@ELK-5 elastalert]# elastalert-create-index
    Enter Elasticsearch host: localhost
    Enter Elasticsearch port: 9200
    Use SSL? t/f:
    Enter optional basic-auth username (or leave blank):
    Enter optional basic-auth password (or leave blank):
    Enter optional Elasticsearch URL prefix (prepends a string to the URL of every request):
    New index name? (Default elastalert_status)
    Name of existing index to copy? (Default None)
    Traceback (most recent call last):
    File “/root/elastalert/virtualenvelastalert/bin/elastalert-create-index”, line 11, in
    load_entry_point(‘elastalert==0.1.6’, ‘console_scripts’, ‘elastalert-create-index’)()
    File “/root/elastalert/virtualenvelastalert/lib/python2.7/site-packages/elastalert-0.1.6-py2.7.egg/elastalert/create_index.py”, line 120, in main
    if es_index.exists(index):
    File “build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py”, line 69, in _wrapped
    File “build/bdist.linux-x86_64/egg/elasticsearch/client/indices.py”, line 225, in exists
    File “build/bdist.linux-x86_64/egg/elasticsearch/transport.py”, line 327, in perform_request
    File “build/bdist.linux-x86_64/egg/elasticsearch/connection/http_requests.py”, line 79, in perform_request
    elasticsearch.exceptions.ConnectionError: ConnectionError(HTTPConnectionPool(host=’localhost’, port=9200): Max retries exceeded with url: /elastalert_status (Caused by NewConnectionError(‘: Failed to establish a new connection: [Errno 111] Connection refused’,))) caused by: ConnectionError(HTTPConnectionPool(host=’localhost’, port=9200): Max retries exceeded with url: /elastalert_status (Caused by NewConnectionError(‘: Failed to establish a new connection: [Errno 111] Connection refused’,)))

    Like

    • Hi meia! From the error, it appears your Elasticsearch is not running or it is not running on localhost:9200. Could you please show your Elasticsearch configuration and check if it is running properly before the command is executed?

      Thank you

      Like

  7. janoo bhandari

    while running “elastalert-create-index” command, Iam getting the following error.

    root@practice1:/home/ubuntu/elastalert# elastalert-create-index
    Traceback (most recent call last):
    File “/usr/local/bin/elastalert-create-index”, line 5, in
    from pkg_resources import load_entry_point
    File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 2749, in
    working_set = WorkingSet._build_master()
    File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 446, in _build_master
    return cls._build_from_requirements(__requires__)
    File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 459, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
    File “/usr/lib/python2.7/dist-packages/pkg_resources.py”, line 628, in resolve
    raise DistributionNotFound(req)
    pkg_resources.DistributionNotFound: elasticsearch<3.0.0

    Could you please help?

    Like

      • Zanu

        hi alexandreesl, ys i tried install with “pip install -r requirements.txt”.

        here’s the output:
        (virtualenvelastalert) [root@localhost elastalert]# sudo pip install -r requirements.txt
        Collecting argparse==1.3.0 (from -r requirements.txt (line 1))
        Using cached argparse-1.3.0-py2.py3-none-any.whl
        Collecting aws-requests-auth==0.2.5 (from -r requirements.txt (line 2))
        Using cached aws-requests-auth-0.2.5.tar.gz
        Complete output from command python setup.py egg_info:
        /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: ‘install_requires’
        warnings.warn(msg)
        usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] …]
        or: -c –help [cmd1 cmd2 …]
        or: -c –help-commands
        or: -c cmd –help

        error: invalid command ‘egg_info’

        —————————————-
        Command “python setup.py egg_info” failed with error code 1 in /tmp/pip-build-BYmYHM/aws-requests-auth/

        Like

      • Hi, Zanu! Have you tried the procedure on the documentation on this link: https://core.telegram.org/bots ?

        You can also use the bot I created just for testing your configuration, all you have to do is enter the channel “Elasticalert hands-on” on Telegram as a member and use the same YAML configuration I provided.

        Thank you!

        Like

  9. Nirav Shah

    Great article, but i have a question :
    I am trying to add
    aggregation:
    schedule: ‘*/5 15-23 * * *’

    I Want to receive an alert(s) every 5 Minutes from 15:00 hrs till 23:00 hours, daily. but in my case, it’s not happening it. I got all the alerts on other times, I was under the impression once I put schedule will trigger the alert(s) during given time. It’s not happening it. Can you please help me?

    Like

  10. Monty

    Hi,

    I have a requirement to create dynamic rules for each user using an Elasticsearch instance queries.Let me know how I can implement this as Elastalert doesn’t seem to work with database for rules settings dynamically

    Like

    • sorry man, my last year was cruel, just saw your comment now. Anyway, indeed, unfortunately, you can’t use elastalert dynamically this way out-of-the-box, the only option on this case as I see is if you fork the project and integrate the alerting engine with your database rules repository. Sorry for being so late, thank you for your visit.

      Like

  11. navn

    My alert are not consistent.I want an alert to come if there is a wrong ssh on my server.Sometime I am getting the alert other time I am not able to.

    Like

  12. Pingback: Reducing and Learning from Monitoring Alerts in Business Environments | Number ONE
  14. annonymos

    In elast alert can we use type: change and compare_key: test1 or test2 (because both fields we would like to compare a change of the value). Can this be possible?

    Like

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.