Technology Blog

• @attdigitaldumbo holla Brooklyn #

• NYC area? We're looking for a full time web developer and a web developer intern…….jobs@creatingdigital.com …. please pass along! #

• Got my @scorpionsport gear on…. Comfortable hoodie with my ringer shirt #heaven #iwannanap #
• Gomigo http://t.co/xR7UqdX #

This week, I’ve set up mysql cluster on virtual machines to do some benchmarking and testing against performance.

For the basis I have implemented the steps outlined here.

I ran into a few snags however.  The first snag was that I was using virtual machines from rackspace cloud with 256MB of ram.  Surely I could run mysql with only 256MB of ram right?  In 7.1 they allowed disk access to the database so I should be fine, right?  – NOPE.


Mar 17 21:01:09 db2 kernel: [11925.786630] ndbd invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
Mar 17 21:01:09 db2 kernel: [11925.786643] ndbd cpuset=/ mems_allowed=0
Mar 17 21:01:09 db2 kernel: [12298.051365] Pid: 1435, comm: ndbd Not tainted 2.6.35.4-rscloud #8


So I upgraded to a 512MB virtual machine – SAME THING. Apparently you need ATLEAST 1GB of ram. As once we upgraded to a 1GB virtual Machine we were able to fix this problem.

Two other issues that can cause this are SELINUX. To check run the below command:


cat /etc/selinux/config |grep ^SELINUX=


This should output: SELINUX=disabled.

Another reason can be in your cluster config.ini file – having the DataMemory and IndexMemory values too high can cause issues. The default settings that worked for me initially (just to start the service) were:

DataMemory=80M
IndexMemory=12M


Here’s something that cost me 2-3 hours of work: I initially had the DataMemory and IndexMemory set TOO high (extra 0) and it was trying to malloc 5GB of ram when only 2 GB of ram were in the machine. This was causing errors like:


2011-03-18 18:14:23 [ndbd] INFO -- NDB Cluster -- DB node 3
2011-03-18 18:14:23 [ndbd] INFO -- mysql-5.1.51 ndb-7.1.10 --
2011-03-18 18:14:23 [ndbd] INFO -- numa_set_interleave_mask(numa_all_nodes) : OK
2011-03-18 18:14:23 [ndbd] INFO -- Ndbd_mem_manager::init(1) min: 5268Mb initial: 5288Mb
ndbd_malloc_impl.cpp:561:grow(111030, 1152) 111030!=0 not using 36Mb - Unable to use due to bitmap pages missaligned!!
2011-03-18 18:14:23 [ndbd] ERROR -- ndbd_malloc_impl.cpp:565:grow(111030, 1152) not using 36Mb - Unable to use due to bitmap pages missaligned!!


I then changed the values in config.ini and re-ran the below thinking it would reload the config, right?:
ndb_mgmd -f /var/lib/mysql-cluster/config.ini

This DID NOT work. So then I tried –reload:
ndb_mgmd -f /var/lib/mysql-cluster/config.ini --reload

I’m smart, I read the man entry…..wtf – this didn’t work again. I’m getting sick of re-booting these machines (thank goodness for VM technology – NOC guys would have hated me right about now).

Seems you need to call it with the –initial flag AGAIN in order to flush the config.

Beginning with MySQL Cluster NDB 6.4.0, configuration data is cached internally rather than being read from the cluster
global configuration file each time the management server is started (see Section 17.3.2, âMySQL Cluster Configuration
Filesâ
and then to re-read the configuration data from the cluster configuration file and to build a new cache.

So I ran
ndb_mgmd -f /var/lib/mysql-cluster/config.ini --initial

And then started ndbd on the node machines and WOOO – it worked!

Hope this article saved somebody an hour or 3! Documentation and thorough articles on this stuff is hard to come by. Leave me a post in the comments field if this helped you.

• congrats to @scorpionsport ( http://bit.ly/gxr7kH ) for the Inside Lacrosse coverage at http://bit.ly/eR7DC8 and http://bit.ly/ftDIKu #

• @alexmbt make sure it has an export feature. It would e a shame for someone else to own all that content! #

• @davidcanter heard you may need a web development company? Is this true. Can we send you our agency overview? #

• The Best Daily Deals in NYC Uptown: Dinner for Two: http://bit.ly/h8hag8 via @addthis #

• “@Stevemini: Wow – HomeAway.com – didn't know they had $3mm for the spot. Crazy ad though. #superbowl #brandbowl” totally agree #
• @stevemini is it just me or has cars.com dropped 20mm today? #
• @stevemini tried homeaway on iPhone before – didn't load – either too much server load or not mobile optimized #fail #
• Verizon hammers AT&T will they have same problems though – only time will tell #superbowl #

• @mediatemple again 24 hours no response…Been holding for the last 30 minutes….I got 15 people in front of me. ticket #1048599 #