In this blog, we will create a python script to get a slack notification when “Out of memory” string is found in the log file.
Step1: Slack Configuration
- First, you need to have a slack account and a workspace.
- Please visit the following link: https://api.slack.com/
- Sign in to your slack account.
- Click on Create an app
- Next, click create from scratch.
- Write an app name and select your workspace.
- Select Bots in Add features and functionality.
- Click Review scopes to add.
- Click Add an OAuth scope in Bot token scopes.
- Select chat:write scope.
- Next, click Install to workspace and select Allow.
- Copy the Bot User OAuth Token.
- Now go to your slack workspace and create a new channel.
- Click Integrations->Add an app.
- Add the app that we created.
Step2: Write the python script
This script will send the slack notification if the “Out of memory” string is detected in the log file. We will use the awk command to get logs of only the last five minutes so that we do not receive notifications for older logs.
import slack
import subprocess
cmd = "sudo awk -v d1=\"$(date --date=\"-5 min\" \"+%b %_d %H:%M\")\" -v d2=\"$(date \"+%b %_d %H:%M\")\" '$0 > d1 && $0 < d2 || $0 ~ d2' /var/log/messages |grep -i 'out of memory'"
sp=subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True)
rc=sp.wait()
if sp.poll() :
pass
else:
stdout,stderr=sp.communicate()
message=stdout
client= slack.WebClient(token='Auth Token')
client.chat_postMessage(channel='#channel-name', text=f"{message}")
- Replace channel name and Auth token in the script.
Step3: Create a cron job
- Next, we will create a cronjob so that our script runs every five minutes.
- Use the following command to open crontab.
sudo crontab -e
- Paste the following command in the crontab.
*/5 * * * * /usr/bin/python3 /home/vagrant/scripts/out-of-memory.py
- To test the script we will use the stress command to increase memory load.
stress --vm 50 --timeout 200s
- Now check your slack channel, you should receive a notification in some time.
Author Information
This is blog is written by Checkmate Global Technologies engineer. Please contact our technical consultants to discuss about product engineering, product development, IT staff augmentation and cloud infrastructure.
