Posts

How to analyze S3 logs and move to Local system.

Amazon web services move there web logs or any type of logs in S3 so in order to analyze them we need to parse them either via any third party application or download them individually and work on it.
Following script will help us to parse S3 files directly and create single file from it.
[root@ip-10-0-1-220 ravi]# cat s3logs.sh #!/bin/bash #By Ravi Gadgil #Script to get latest logs from S3 and sync to System log. echo -e "\n------------------------------\n`date`\n-----------------------------" s3location=$(echo -e "s3://teamie-logs/S3Logs") echo -e "The S3 Bucket location is : $s3location" #To check if its first time sync if [[ ! -f /tmp/lastfile.txt ]]; then #To get last 200 files names sorted by date. aws s3 ls $s3location/ | tail -n 200 | awk '{print $4}' > /tmp/filesync.txt for i in `cat /tmp/filesync.txt`; do echo "Following log file is being synced : $i " aws s3 cp $s3location/$i - >> /var/log/s3hits.log done #To …

How to analyze AWS RDS Slow Query

Amazon RDS has the feature in which we can log the slow queries and than analyze them to find the queries which are making our Database slow.
There are few prerequisites to this:
1. Slow log query should be enabled in RDS.
2. AWS RDS CLI should be present.
3. Percona Toolkit should be present.

Once you have all the prerequisites than following script will help us in rest.
[root@ip-10-0-1-220 ravi]# cat rds-slowlog.sh #!/bin/bash #Script to Analyze AWS RDS MySQL logs via Percona Toolkit #By Ravi Gadgil #To get list of all slow logs available. /opt/aws/apitools/rds/bin/rds-describe-db-log-files --db-instance-identifier teamie-production --aws-credential-file /opt/aws/apitools/rds/credential | awk '{print $2 }' | grep slow > /home/ravi/slowlog.txt logfile=$(echo -e "slowlog-`date +%F-%H-%M`") resultfile=$(echo -e "resultlog-`date +%F-%H-%M`") for i in `cat /home/ravi/slowlog.txt` ; do #To download Slow Log files and add them to single file. /opt/aws/ap…

How to install Percona MySQL tools

Persona has lots of tools to analyses MySQL data and very useful information can be extracted from them.

Installing via Yum:
[root@ip-10-0-1-220 ravi]# yum install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm Retrieving http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm Preparing... ########################################### [100%] 1:percona-release ########################################### [100%] [root@ip-10-0-1-220 ravi]# yum install percona-toolkit Loaded plugins: auto-update-debuginfo, priorities, update-motd, upgrade-helper 1123 packages excluded due to repository priority protections Resolving Dependencies --> Running transaction check ---> Package percona-toolkit.noarch 0:2.2.18-1 will be installed --> Processing Dependency: perl(DBD::mysql) >= 1.0 for package: percona-toolkit-2.2.18-1.noarch --> Processing Dependency: perl(DBI) >= 1.13 for…

How to setup AWS RDS CLI

AWS provides the its CLI tools but few of RDS functionalities don't work on them so in order to make them work AWS RDS CLI is very helpful.

Download the RDS CLI:
[root@server downloads]# wget http://s3.amazonaws.com/rds-downloads/RDSCli.zip
Unzip the downloaded file and copy it in desired location:
[root@server downloads]# unzip RDSCli.zip[root@server downloads]# cp -r RDSCli-1.19.004 /opt/aws/apitools/rds [root@server downloads]# cd /opt/aws/apitools/rds/bin
Check RDS CLI version:
[root@server bin]# ./rds-version Relational Database Service CLI version 1.19.004 (API 2014-10-31)
These commands can be added in system environment so you can run them from anywhere:
[root@server bin]# export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/apitools/rds/bin:/usr/local/bin
The credential file is also need to be created so that these commands can be run taking permissions in to consideration from that file and file can be added in system environment so we don't need to pass every time in co…

Script to add logs in Logentires in case of Shared hosting Web Server.

Logentries is a very good tool to store and analyze logs in central location but when we are storing logs from a  Nginx/Apache shared hosting environment it gets complex as we need to tag each log to which host it belong. I am using rsyslog to forward to send my logs to Logentries as it gives me more flexibility as all rsyslog functionality works.

First need to create a main configuration file which we will use to create corresponding rsyslog files. It will have your Logentries secret key to whom log need to be followed.

For access log:
[root@ip-10-0-1-220 ravi]# cat access-vanila $Modload imfile $InputFileName access-log-location $InputFileTag access-tag $InputFileStateFile filestate-tag $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor # Only entered once in case of following multiple files $InputFilePollInterval 1 $template filestate-tag,"6fb8xxxxxxxxxxxxxxxxxxxxxxe8ed %HOSTNAME% %syslogtag% %msg%\n" if $programname == 'access-tag' then @@…

How to install python2.7 with pip2.7

Python is one of the most famous and powerful languages used so in order to install or update it to 2.7 version following steps can be used.


Installing the Python2.7.
[root@ip-10-0-1-55 ~]# yum install python27 python27-devel[root@ip-10-0-1-55 ~]# python --version Python 2.7.10
If its still showing older version add the new version as default.
[root@ip-10-0-1-55 ~]# alternatives --config python There are 2 programs which provide 'python'. + 1 /usr/bin/python2.6 * 2 /usr/bin/python2.7 Enter to keep the current selection[+], or type selection number: 2 [root@ip-10-0-1-55 ~]# python --version Python 2.7.10
Installing the pip2.7.
[root@ip-10-0-1-55 ~]# wget https://bootstrap.pypa.io/get-pip.py --2016-05-26 11:10:33-- https://bootstrap.pypa.io/get-pip.py Resolving bootstrap.pypa.io (bootstrap.pypa.io)... 103.245.222.175 Connecting to bootstrap.pypa.io (bootstrap.pypa.io)|103.245.222.175|:443... connected. HTTP request sent, awaiting response... 200 OK Length:…

How to create and remove swap partition in Linux

If our server is having memory issues and we want to increase it without updating the physical RAM than Swap is a very good option. It's slower than the physical RAM but can do the tasks also if created on faster hard disks the results are good.

Create a Swap partition in place where you wanna create it. I am creating it in /swap with 2GB of size.
[root@ip-10-0-1-38 /]# dd if=/dev/zero of=/swap bs=1M count=2048 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 54.8004 s, 39.2 MB/s
Update the permissions of /swap partition to 600.
[root@ip-10-0-1-38 /]# swapon /swap swapon: /swap: insecure permissions 0644, 0600 suggested. [root@ip-10-0-1-38 /]# chmod 600 /swap
Start the Swap on /swap partition.
[root@ip-10-0-1-38 /]# swapon /swap
To check if swap has started or not.
[root@ip-10-0-1-38 /]# swapon -s Filename Type Size Used Priority /swap file 2097148 0 -1 [root@ip-10-0-1…