Technology Blog

So we use check_mk pretty extensively for all of our servers for health monitoring. Today we added a brand new bare metal centos 7 minimal install on-sit test server to nagios.

We were getting this strange service when we added the box to be invetnoried:

Resolve_mathias-kettner.de
CRITICAL 06-29-2015 13:43:49 3d 5h 14m 51s 3/3 CRIT - mathias-kettner.de could not be resolved

I couldn’t figure out what this check does so I took a peak at the git repo, and saw that its doing a nslooup.  Well my box didn’t have nslookup installed, so a quick

sudo yum install bind-utils -y

seem to do the trick!

I saw the check_mk added the ability to monitor the qmail mail queue, so I wanted to get that setup on a few plesk servers. The auto inventory wasn’t seeing the qmail and therefore wasn’t adding the check. It turns out that two things were wrong:

• I updated the check_mk agent on the plesk server, but didn’t update the check_mk software on the nagios host. I needed to install the latest version in BOTH places to get it working.
• Plesk uses its own paths, so we had to symlink /var/qmail/bin/qmail-qstat to /bin/qmail-qstat

ln -s /var/qmail/bin/qmail-qstat /bin/qmail-qstat

I spent way too much time figuring that out. Hopefully this post will help the next poor soul!

This shell script finds all the tables for a database and run a command against it.

See File optimize-tables

The file’s contents are below as well:

#!/bin/sh

# this shell script finds all the tables for a database and run a command against it
# @usage "mysql_tables.sh --optimize MyDatabaseABC"
# @date 6/14/2006
# @version 1.1 - 1/28/2007 - add repair
# @version 1.0 - 6/14/2006 - first release
# @author Son Nguyen

DBNAME=$2

printUsage() {
  echo "Usage: $0"
  echo " --optimize <dbname>"
  echo " --repair <dbname>"
  return
}


doAllTables() {
  # get the table names
  TABLENAMES=`mysql -D $DBNAME -e "SHOW TABLES\G;"|grep 'Tables_in_'|sed -n 's/.*Tables_in_.*: \([_0-9A-Za-z]*\).*/\1/p'`

  # loop through the tables and optimize them
  for TABLENAME in $TABLENAMES
  do
    mysql -D $DBNAME -e "$DBCMD TABLE $TABLENAME;"
  done
}

if [ $# -eq 0 ] ; then
  printUsage
  exit 1
fi

case $1 in
  --optimize) DBCMD=OPTIMIZE; doAllTables;;
  --repair) DBCMD=REPAIR; doAllTables;;
  --help) printUsage; exit 1;;
  *) printUsage; exit 1;;
esac

Important Note:

Since these were Rackspace servers – the default repository that they connect to does not have the xen kernel packages and xen software on them.  We had to call and have Rackspace add the capability of using certain commands like “yum install xen”

Step 1.1:Install the xen kernel and xen.

# yum --disableexcludes=all install kernel-xen xen

Step 1.2: Reboot

# reboot

Step 1.3:Ensure the new kernel loaded.  You should see xen after the kernel version:

# uname –a

Linux web1.myhostname.com 2.6.18-238.9.1.el5xen #1 SMP Fri Mar 18 12:53:42 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

Step 1.4:Ensure xen is running on boot.

# xm list

Name                                      ID Mem(MiB) VCPUs State   Time(s)

Domain-0                                   0     7989     8 r-----   6268.9

 

Congrats the host machine is setup!

Step 2: Let’s setup a CentOS virtual machine.  Virt-install:

# virt-install -–prompt

What is the name of your virtual machine? mnode1

How much RAM should be allocated (in megabytes)? 8024

What would you like to use as the disk (file path)? /home/xen/mnode1

How large would you like the disk (/home/xen/mnode1) to be (in gigabytes)? 100

What is the install URL? http://mirror.cogentco.com/pub/linux/centos/5.6/os/x86_64/

 

<code>Starting install...</code>

Retrieving file .treeinfo...                             |  417 B     00:00

Retrieving file vmlinuz...                               | 2.1 MB     00:01

Retrieving file initrd.img...                            | 7.9 MB     00:03

Creating storage file...                                 | 100 GB     00:00

Creating domain...                                       |    0 B     00:05

error: Failed to get local hostname

Domain installation still in progress. You can reconnect to

the console to complete the installation process.

Step 4:At this point you need to connect to the virtual machines “console” to continue.

# xm console <hostname>

Step 5:Go through the centos install…..I’ll omit this process.  But in short you WILL need to know the ip address, subnet, and gateway of your server.

Install iptables (Hopefully, you’ve done this already)

# yum install iptables* -y

Download, unpack, install of APF from source:

# cd /usr/local/src

# wget http://www.rfxn.com/downloads/apf-current.tar.gz

# tar -zxf apf-current.tar.gz

# cd apf-9*

# ./install.sh

Cleanup source install files:

# rm -Rf /usr/local/src/apf-9* && cd

Backup original APF config:

# cp /etc/apf/conf.apf /etc/apf/conf.apf.bak

Edit APF Configuration file:

# nano -w /etc/apf/conf.apf

Change the following settings:

RAB="0" to RAB="1"

RAB_PSCAN_LEVEL=”2″ to RAB_PSCAN_LEVEL=”3″

TCR_PASS=”1″ to TCR_PASS=”0″

DLIST_PHP=”0″ to DLIST_PHP=”1″

DLIST_SPAMHAUS=”0″ to DLIST_SPAMHAUS=”1″

DLIST_DSHIELD=”0″ to DLIST_DSHIELD=”1″

DLIST_RESERVED=”0″ to DLIST_RESERVED=”1″

IG_TCP_CPORTS=”21,22,25,53,80,443,3306″

IG_UDP_CPORTS=”″

EGF=”1”

EG_TCP_CPORTS=”22,3306,80,443″

EG_UDP_CPORTS=”″

DEVEL_MODE=”0”

Step 20: Reload APF:

# /usr/local/sbin/apf–r

Step 21:Make sure APF starts automatically after reboot:

# chkconfig --add apf

# chkconfig –level 345 apf on

This week, I’ve set up mysql cluster on virtual machines to do some benchmarking and testing against performance.

For the basis I have implemented the steps outlined here.

I ran into a few snags however.  The first snag was that I was using virtual machines from rackspace cloud with 256MB of ram.  Surely I could run mysql with only 256MB of ram right?  In 7.1 they allowed disk access to the database so I should be fine, right?  – NOPE.


Mar 17 21:01:09 db2 kernel: [11925.786630] ndbd invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
Mar 17 21:01:09 db2 kernel: [11925.786643] ndbd cpuset=/ mems_allowed=0
Mar 17 21:01:09 db2 kernel: [12298.051365] Pid: 1435, comm: ndbd Not tainted 2.6.35.4-rscloud #8


So I upgraded to a 512MB virtual machine – SAME THING. Apparently you need ATLEAST 1GB of ram. As once we upgraded to a 1GB virtual Machine we were able to fix this problem.

Two other issues that can cause this are SELINUX. To check run the below command:


cat /etc/selinux/config |grep ^SELINUX=


This should output: SELINUX=disabled.

Another reason can be in your cluster config.ini file – having the DataMemory and IndexMemory values too high can cause issues. The default settings that worked for me initially (just to start the service) were:

DataMemory=80M
IndexMemory=12M


Here’s something that cost me 2-3 hours of work: I initially had the DataMemory and IndexMemory set TOO high (extra 0) and it was trying to malloc 5GB of ram when only 2 GB of ram were in the machine. This was causing errors like:


2011-03-18 18:14:23 [ndbd] INFO -- NDB Cluster -- DB node 3
2011-03-18 18:14:23 [ndbd] INFO -- mysql-5.1.51 ndb-7.1.10 --
2011-03-18 18:14:23 [ndbd] INFO -- numa_set_interleave_mask(numa_all_nodes) : OK
2011-03-18 18:14:23 [ndbd] INFO -- Ndbd_mem_manager::init(1) min: 5268Mb initial: 5288Mb
ndbd_malloc_impl.cpp:561:grow(111030, 1152) 111030!=0 not using 36Mb - Unable to use due to bitmap pages missaligned!!
2011-03-18 18:14:23 [ndbd] ERROR -- ndbd_malloc_impl.cpp:565:grow(111030, 1152) not using 36Mb - Unable to use due to bitmap pages missaligned!!


I then changed the values in config.ini and re-ran the below thinking it would reload the config, right?:
ndb_mgmd -f /var/lib/mysql-cluster/config.ini

This DID NOT work. So then I tried –reload:
ndb_mgmd -f /var/lib/mysql-cluster/config.ini --reload

I’m smart, I read the man entry…..wtf – this didn’t work again. I’m getting sick of re-booting these machines (thank goodness for VM technology – NOC guys would have hated me right about now).

Seems you need to call it with the –initial flag AGAIN in order to flush the config.

Beginning with MySQL Cluster NDB 6.4.0, configuration data is cached internally rather than being read from the cluster
global configuration file each time the management server is started (see Section 17.3.2, âMySQL Cluster Configuration
Filesâ
and then to re-read the configuration data from the cluster configuration file and to build a new cache.

So I ran
ndb_mgmd -f /var/lib/mysql-cluster/config.ini --initial

And then started ndbd on the node machines and WOOO – it worked!

Hope this article saved somebody an hour or 3! Documentation and thorough articles on this stuff is hard to come by. Leave me a post in the comments field if this helped you.

Cpanel Log Location Poster

Coming from a plesk background, I always have a hard time finding the location of Cpanel’s log’s.  Today while looking for where cpanel keeps it’s access logs I stumbled across this poster now available by cpanel.

By the way the location for the apache logs is:
/usr/local/apache/logs/{DOMAIN}


So today I ran into an svn problem: Svn: Path ‘somedirectory’ not present  I spend a good hour trying to fix it.  It turned out to be left over garbage from moving folders around – something I wish svn was better able to do.  You can move folders around fluidly without causing havoc on your repo.

So Follow me I had a structure like this:

site.com

-www

-includes

-js

-css

I wound up renaming the www folder to “sliced” and created a new www folder.  I installed wordpress and did everything I needed to do.  I went to check it in and bamn – got the above error.  After trying several things online – with no good direction I realized all I had to do was create a folder includes that contained js and css folders and an add and checkin:

mkdir -p includes/css

mkdir -p includes/js

svn add includes

svn ci -m "my error is gone"

svn remove includes

svn ci -m "now my extra empty folder is gone as well"

A colleague of mine recently came over to discuss HTML 5 the next major revision of HTML! He wondered what new and beneficial features were in store. The few noticeable ones are:

• New “nav” and “footer” elements
• Canvas tag for immediate bitmap rendering – this is so cool!  No more using gd libraries on servers – use the clients web browser!
• Drag and Drop capability.  If you’ve ever created a website that had users uploading files we know this is a HUGE feature!  Gmail attachments will be so much easier now too!
• Cross Document Messaging – not too much discussion on this one – are two pages going to be communicate with each other?
• Timed media playback.
• Video tag allows in browser playback – cool in theory but they dropped a specific codec, so now this is going to get messy!  Users will need to have codec’s installed or users uploading will need to take care!

These are the highlights.  In my opinion the drag and drop is by far going to be the most anticipated and highly used feature – next to video.  But regular end users won’t notice the video tag as they are use to flash.  So many users won’t even notice a difference.

Unfortunately, we’ll probably have about 10 years to go until we’re able to use these features.  Microsoft IE 6 is still lingering around, so we have to wait for that beast to die (2014)…..and with some hope and luck, we’ll be happy coding in html 5 within 10 years!

Grub Booting

The servers I work on are almost never in front of me. Instead they are in dark rooms humming away gracefully. When I need to do a kernel upgrade or recompile I need to reconfigure grub to reboot into the new kernel. So normally you would just edit /etc/grub.conf and change –save-default=X to be the correct number.

Well what happens if you change that and your new kernel doesn’t boot? Your left black screened with no one looking at the server to tell you what’s wrong with it. This leaves you calling the data center (or your host) asking for a technician to take a look at the server, at the very least reboot login to single user mode, edit the config, and then restart into the old kernel. Sounds time consuming huh?

Yup, and then you still have to figure out what you did wrong, reboot, and pray again. With more and more host offering instant reboots in their control panels, you could do this whole operation without ever bothering anyone at the data center.

Instead of changing /etc/grub.conf just type in the following at the command line:

grub
savedefault --default=X --once
quit

X of course being the number kernel you would like to boot into.

Now if your server doesn’t boot properly into the new kernel you only need to power cycle it to get it to boot into the old working kernel. Host doesn’t offer power cycling? At the very least the data center staff will appreciate the joy of not having to log into your server and fix things. They are usually much quicker at handling simple reboot request than configuration changes. If they are busy they can at least have the janitor go over and power cycle the server, instead of someone who is a grub expert! Point taken?