How to install EC2 API tools

In my previous post I had mentioned that I will show you how to install EC2 API command line tools. This post will guide you to install EC2 tools. This is quite an easy task. Let me show you the steps

1. First you have to download the EC2 API command line tools from Amazon
2. unzip the EC2 API tools
unzip EC2-api-tools.zip

3. Now you have to set up some environment variable

export EC2_HOME=/home/lib/ec2-api-tools
(here I have unzip the EC2-api-tools to home/lib/ec2-api-tools)

export EC2_PRIVATE_KEY=/home/.ec2/pk-XXXXXXXXXXXXXXXXXXXX.pem
(this will be the path to your private key file)

export EC2_CERT=/HOME/.ec2/cert-XXXXXXXXXXXXXXXXXXXX.pem
(this will be the path to your cert file)

Note: you can directly put these lines to you ~/.bashrc file. Another thing you have to remember that EC2-command-line tools use JAVA so you have to install JAVA on the machine and have to set the JAVA_HOME path as well

Integrating EBS(Elastic Block Storage)

Amazon AWS provides really strong support to make the application more scalable and secured. EBS is one of that support what Amazon provide.

The main two main supports that EBS will provide you as following

1. Incremental database back up

2. EBS will take the snapshot and upload it to S3. so somehow if your EC2 instance get destroyed then you can attach the volume to another EC2 instance and restore the snapshot.

Few days back I was thinking how can I take an incremental back up for a project. And thank god that our project was deployed on EC2 and Amazon has provided the EBS support. Now lets have a look how can we integrate EBS. Here I will also discuss some problem at the end what I have faced to integrate EBS.

1. first we have to create a volume for this reason you will need to download and configure EC2 command line API tools (in my next blog i will show you how to install it)
   

          ec2-create-volume -z us-east-1b -s 10  
      (here –z means zone, remember that this should be same for instance and volume
      and –s means the size of the EBS storage in GB)

NOTE: this command will return the volume number of newly created volume take a note of that volume number , we have use it later.

2. You can give the following command to view the details of the volume.

        ec2-describe-volumes vol-XXXXXXXX
      (this number should be  the returned number of the previous command)

3. Now you have to attach the volume with the EC2 instance from which you want to take the data base backup

      ec2-attach-volume -d /dev/sdh -i i-XXXXXXX vol-XXXXXXXX 

       (here i-XXXXXXX is the instance number and vol-XXXXXXXX is the volume number)

4. you have to log in into the EC2 instance
   

5. now you have to use the following commands to install xfs which will help you to take the back up incrementally
   

               apt-get update && apt-get upgrade -y
               apt-get install -y xfsprogs mysql-server

6. now you have to create XFS file system and mount a directory from where volume will take the database updates for the snapshot
   

mkfs.xfs /dev/sdh
echo "/dev/sdh /vol xfs noatime 0 0" >> /etc/fstab
mkdir /vol
mount /vol

7. now we have to configure MySQl to use the EBS volume
   

1. stope the MySQL Server
   

                  /etc/init.d/mysql stop

2. Move the existing database files to the EBS volume (My recommendation is to copy the data base it will be safe if you face any problem then you can come to current position very quickly and this is risk free)
   

               mkdir /vol/lib /vol/log
               mv /var/lib/mysql /vol/lib/   
             (my recommendation : cp –rf /var/lib/mysql/* /vol/lib/ )
               mv /var/log/mysql /vol/log/
             (my recommendation : cp –rf /var/log/mysql/* /vol/log/ )

test -f /vol/log/mysql/mysql-bin.index &&
  perl -pi -e 's%/var/log/%/vol/log/%' /vol/log/mysql/mysql-bin.index

8. now we have to tell MySQL to look at on the mounted EBS volume and also save the configuration of EBS for future use

cat > /etc/mysql/conf.d/mysql-ec2.cnf << EOM
[mysqld]
innodb_file_per_table
datadir          = /vol/lib/mysql 
log_bin          = /vol/log/mysql/mysql-bin.log
max_binlog_size  = 1000M
log_slow_queries = /vol/log/mysql/mysql-slow.log
long_query_time  = 10
EOM

Note: In my case I have to add tmpdir = /vol/tmp because I have shifted that folder on EBS.

        rsync -aR /etc/mysql /vol/

9. now you have to start the mysql

         /etc/init.d/mysql start

Note: if MySQL gets started successfully then you can remove the previous data directory of our server

10. now you have to take the snapshot for the mysql backup
1. first you have to log in into mysql using your password and user name
   

mysql -uusername –ppassword

2. flush the tables to disk and acquire a lock. Flush the file system to disk and freeze it. Do not exit the MySQL session or you will lose the lock and snapshot potentially inconsistent database files
   

FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;
SYSTEM xfs_freeze -f /vol

3. now you have to create the snapshot of the current volume from you pc using ec2 command line tools
   

                  ec2-create-snapshot vol-XXXXXXXX 

4. Back in the same MySQL session on the instance, unfreeze the file system, release the database lock, and you're done
   

                     SYSTEM xfs_freeze -u /vol
                     UNLOCK TABLES;
                     EXIT               

You can describe the snapshot to see the status and to know whether it is done or not.

                  ec2-describe-snapshots snap-xxxxxx