How to setup an Alarm when RDS is running on low free disk space

Yea, that happened to me!

The Problem

My RDS instance suddenly ran out of space and some of our applications started failing left, right and centre. It was a disaster and a fair bit of fire fighting was involved.

I told to myself, how did this happen? I should have put checks in place to ensure this didn't happen. I should have added some sort of alarm to warn when free disk space is low.

To deal with this, we wanted to first setup an alarm to notify the team when RDS instance is running on low free disk space. Looked at AWS console to create the alarm, but - I must admit - we were a bit surprised to see that there isn't like a straightforward way to create this type of an alarm.

The Solution

After a little googling, we found the way to setup the Alarm. This post is to document the steps involved in getting this done so that, I do not forget them :D

We basically need to do the following

• Create an SNS topic to that can send emails

• Subscribe the team email address to the SNS topic

• Confirm the email subscription by clicking on the link that AWS sends.

• Create a cloud watch alarm to send the alert when the RDS free disk space is lesser than the chosen threshold

That's all there is to it!

How to restart AWS Elastic BeanStalk instances on a schedule

We wanted to restart all AWS Elastic BeanStalk instances at a given time during the day, every day. To perform this task, there are many solutions. Some of them involve, setting up a lambda function and writing some code etc, what we wanted was something quick and not a maintenance nightmare.

Hence, I thought, all we need to do is re-start the instance on a schedule, whats the best way to do it? Cron jobs flashed as a possible answer!

The Problem

The environment we were talking about was an Elastic BeanStalk Auto-Scaling environment, which meant that EC2 instances will be added and removed on demand.

If we were going to use cron jobs, we needed to make sure that whatever instances are currently in service, they all honour the cron job at all times. This means, new instances that get added on demand, in service should automatically have the cron jobs setup on them too!

The Solution

To do this in the easiest possible way, we ended up using the Advanced Environment Customization with Configuration Files (.ebextensions). AWS Elastic BeanStalk has a feature where we could provide configuration files bundled with our web application source code. These files are basically configuration files under the directory named ".ebextensions".

Now, to setup the cron job reliably on Elastic BeanStalk instances, all we needed to do was to add the "cron-linux.config" file under the ".ebextensions" directory and bundle it with the application source code which gets deployed to the environment.

The folder structure would look like this

The cron-linux.config will be YAML file. Here are sample contents of the cron-linux.config file, this setup re-starts the Elastic BeanStalk instance every day at 0530 hrs.

Thats about it, deploying this along with you source bundle on the environment ensures that the environment instances are restarted every day at 0530 hrs without fail!

How to get Snowplow-Mini running on AWS

While looking at various Analytics engines we came across Snowplow Analytics. We wanted to give it a shot and experience it first hand. Luckily, they have something called as Snowplow-Mini. Its an easily deployable, single instance version of Snowplow. It essentially gives us, a taste of what Snowplow can do for us, as far as data collection, processing and analytics is concerned!

We started with the quick start guide and usage guide, performed all the steps mentioned there to get the Snowplow-Mini instance working. However, we did faced two annoying issues, investigating and fixing them, wasted a few hrs. This post is about those two issues, so that my fellow developers do not have to waste any time on investigating and fixing them.

Unable to: Generate a pair of read/write API keys for the local Iglu schema registry

We followed all steps mentioned in the usage guide but we were unable to generate the keys.

• Navigate to http://<public dns>/iglu-server
• Input the super API key set up in previous step, in the input box in the top right corner
• Expanded the keygen section
• Expanded the POST /api/auth/keygen operation
• Input the appropriate vendor_prefix for this API key
• Click Try it out!

At this, it should have generated the read and write keys for us. But all it did instead was, showed a progress bar and runs forever without return.

Investigating it in Chrome Developer Console revealed that the calls were failing with 401 UnAuthorized. After googling for this error a bit, I found that someone else was also facing a similar problem. Their solution was to do HTTP POST via CURL and that seemed to work. However it didn't work for us either.

I looked around for ways to debug the problem.

• I connected to the Showplow-Mini instance via SSH (refer to AWS documentation on how to do this)
• Checked the config under "snowplow" directory on the instance. Could not spot anything unusual there -- not that I knew much about it anyways :D
• Checked the logs under "/var/logs" directory. Found a few things but could not really solve the problem.
• Connected to PostgreSQL DB on the instance using the following command
• psql --host=localhost --port=5432 --username=snowplow --dbname=iglu
  # Password is "snowplow"
• Ran the query to check the API key
• select * from apikeys;
• What I saw next, made my jaw drop, in disbelief!
• They API key is case-sensitive and the key Snowplow-Mini had saved was all in lowercase, even though when I had given it the key, I had given it in all caps.

• Passing the key in small case and making the following call did result in generating the read/write API keys for local iglu schema registry
• curl http://<IP address of your server>/api/auth/keygen -X POST -H "apikey: <your case sensitive API key>" -d "vendor_prefix=com.makkajai"
• Duh! Yea I know.

• How to connect to PostgreSQL Snowplow-Mini DB, I got to know that from here 

I must have easily wasted an hour trying to fix this problem. I hope others can save that time!

Unable to: See events in Kibana Dashboard

This was a tricky one. After raising sample events, I was unable to see them in Kibana Dashboard. This happens mainly because the "snowplow_stream_enrich" is not able to connect to the "elastic search service".

How Did I figure it out?

• ssh into the Snowplow-Mini instance
• I checked the logs under "/var/logs" directory. 
• The logs seemed to be filled with exceptions like
• Exception in thread “main” java.net.UnknownHostException: ip-xx-xx-xx-xx: ip-xx-xx-xx-xx: unknown
• Googled it a bit, found the solution here 
• Edit the file "/etc/hosts" and add the IP address information in that file as follows.
• sudo vim /etc/hosts 
• xx.xx.xx.xx ip-xx-xx-xx-xx localhost
• xx.xx.xx.xx being the AWS local IP address.
• Save and exit and re-start all services from the Snowplow-Mini console.
• Generate a few events and open Kibana dashboard, and it worked this time!

After these two problems were out of the way, my Snowplow-Mini instance was fully up and running on AWS!