XenServer APC Smart-UPS shutdown and monitoring
15 04 2011My Citrix XenServer pool is powered through network enabled APC Smart-UPS (6000 RT)
There are at least two ways of monitoring UPS state in order to shutdown pool members and VMs running in the pool when juice is over:
– apcupsd (installation used to be somewhat tricky before XenServer 5.6 and included some unsupported movements)
– getting values from UPS via SNMP and acting accordingly to them – the way which I went, and you’re welcome to see detailed route description.
First of all you’ll need to install snmpget utility on your pool master, it is available from standard repo:
# yum install net-snmp-utils
We will use following command in crontab to run our monitoring script:
* * * * * root /root/scripts/ups/ups_state_check
here’s ups_state_check itself:
#!/bin/bash #UPS settings HOST="10.10.100.2" COMMUNITY="public" #minimal voltage sufficient for your UPS model to not switch to batteries VOLTS_MIN=160 #minimal runtime you need to properly shutdown all VMs and Xen hosts MINUTES_MIN=15 #administrator email (better if it's crossposted to sms in case shutdown is initiated) EMAIL="admin@somedomain.tld" SNMPGET=`which snmpget 2>/dev/null` SNMPOPTS="-Ov -Oq -v1 -r10 -t2 -c ${COMMUNITY} ${HOST}" #Input voltage OID OID_VOLTS_IN=".1.3.6.1.4.1.318.1.1.1.3.2.1.0" #Battery runtime remaining OID OID_RUNTIME="1.3.6.1.4.1.318.1.1.1.2.2.3.0" VALUE_VOLTS_IN=`${SNMPGET} ${SNMPOPTS} $OID_VOLTS_IN` VALUE_RUNTIME=`${SNMPGET} ${SNMPOPTS} $OID_RUNTIME` #SNMP runtime value reported as D:H:M:S.00 #let's convert it into minutes for convenience OFS="$IFS" IFS=":" arr=( $VALUE_RUNTIME ) IFS="$OFS" DAYS=${arr[0]} HOURS=${arr[1]} MINUTES=${arr[2]} MINUTES_LEFT=$(($DAYS*24*60+$HOURS*60+$MINUTES)) if [ $VALUE_VOLTS_IN -lt $VOLTS_MIN ] then echo "Input voltage is too low ($VALUE_VOLTS_IN). $MINUTES_LEFT minutes of UPS runtime remaining." | mail -s "UPS: Power failure report" $EMAIL fi if [ $MINUTES_LEFT -lt $MINUTES_MIN ] then echo "Achtung! $MINUTES_LEFT minutes of UPS runtime remaining - shutdown initiated." | mail -s "UPS: Xenserver pool shutdown initiated." $EMAIL #let the message pass through: sleep 3 #xe_shutdown_vms_and_hosts fi |
and finally the xe_shutdown_vms_and_hosts script wich does actual shutdown of xenserver pool and everything in it:
#!/bin/bash ###enumerate uuid's of all _running_ VMs in the pool for VM in `xe vm-list is-control-domain=false power-state=running params=uuid | egrep -o "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"` do ###(uncomment following line to perform actual shutdown) #xe vm-shutdown vm=$VM echo $VM > /dev/null done ###get uuid of pool master - it must be the very last to shut down MASTER_UUID=`xe pool-list params=master | egrep -o "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-0-9]{4}-[a-f0-9]{12}"` ###enumerate of all xen hosts in the pool except master for HOST in `xe host-list params=uuid | egrep -o "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"` do if [ $HOST != $MASTER_UUID ] then ###(uncomment following line to perform actual shutdown) #xe host-shutdown host=$HOST echo $HOST > /dev/null fi done ###finally shutdown pool master ###(uncomment following line to perform actual shutdown) #xe host-shutdown host=$HOST |
No need for egrep, you can use –minimal to get a comma separated list, for example
xe vm-list is-control-domain=false power-state=running params=uuid –minimal
Then just add the following to the top of the script:
IFS=”,”
I had to make a few changes to the script to get it working for me.
#!/bin/bash
IFS=”,”
###enumerate uuid’s of all _running_ VMs in the pool
for VM in `xe vm-list is-control-domain=false power-state=running params=uuid –minimal`
do
###(uncomment following line to perform actual shutdown)
xe vm-shutdown vm=$VM
echo $VM > /dev/null
done
###get uuid of pool master – it must be the very last to shut down
MASTER_UUID=`xe pool-list params=master –minimal`
###enumerate of all xen hosts in the pool except master
for HOST in `xe host-list params=uuid –minimal`
do
if [ “$HOST” != “$MASTER_UUID” ]
then
###(uncomment following line to perform actual shutdown)
xe host-disable host=$HOST
xe host-shutdown host=$HOST
echo $HOST > /dev/null
fi
done
###finally shutdown pool master
###(uncomment following line to perform actual shutdown)
xe host-disable host=$HOST
xe host-shutdown host=$HOST