Yikes.. Backing up a sharded MongoDB is no fun!

Backing up databases has never been fun, not as fun as having a cool English Ale on the balcony on a hot summar day anyway, but MongoDB takes this one step further when it comes to annoyances.

In general, I often feel that many Open Source projects start with good intentions for what the project should do and how, but then more stuff is added as you require it, and suddenly what started out as a simple and fast application for a narrow usecase, suddenly turns into a bit of a mess. And the issue might well be that building fast, compact software for a specialized usecase, as they start out, is not the same as writing generic software, with a wide range of use cases, code that can easily be maintained and enhanced as we go along. And why should it not be like that? In many cases, this is just fine and the limited usecase is just what the project sets out to do, and it does it well. But sometimes this turns into something really annoying, and at the same time useful.

I think MySQL is partly like this. There are many things in MySQL that work really well, so many things that in a small set of code achives so much useful stuff. And then there are things with MySQL that are just outright wrong and yet another bunch of things that are just plainly annoying. Largely, MySQL is very much developer focused more than DBA focused, although this is improving (and this is also a personal opinion of mine).

And then we have MongoDB, one of the big contenders on the NoSQL side in the NoSQL vs. SQL battle (which is a silly battle, but lets ignore that for now). Now, MongoDB is supposed to be a database. One that is faster, more compact and more targeted towards general database needs than MySQL. A database that can scale and replicate and shards automatically! Brilliant. And then this boring old DBA comes around with his bitterness and boredom and ignorance of the "new" database system. And he says things like: "Can you do a backup"... Yikes! Never thought of that. And what boooring guy that DBA is!

Yes, with MongoDB, backups is clearly an afterthought. And although this is again my personal opinion, I base it on something, namely this: Backing up a sharded Cluster. This is just plain silly. The way these steps are taken is in no way consistent, some of the operations are asynchronous, which means you have to wait for them (write code. To do a friggin' backup? Who came up with THAT daft idea?). You have to backup config servers cold, i.e. shut down. Dead. Who came up with that? And in the end, you don't even get a consistent backup. And yes, if you use replicas, you have to physically back up the replicas also! What?

Whoever figured out how to shard and replicate with MongoDB did a reasonable job, it actually works OK. But the person in question apparently forgot that databases are to be backed up. And before you ask: No, I am NOT going to take a mongodump of 1.5 Tb data!

This said, most aspects of MongoDB are OK, but backups are a mess. Read the page I linked to above, and you also realize that it is not well documented, to say the least, how to backup and what happens if the steps aren't followed? What happens if I do not backup the stopped config server? Can I do a mongodump of the config server instead? Why in heavens name can't I:

• Flush and lock the config servers?
• Flush all the dataservers in one go? No, you have to do it in one dataserver at the time.
• Flush and lock the config servers?
• And yes, why can't you flush and lock the config servers.

Also, locking is weird here. You have can lock serveral times, and then you have to unlock as many times. I really do not know why. Also, I have yet to figure out how to know if it is locked, without at the same time unlocking, but it might be that I missed something in the docs (which aren't very good I'm afraid, but this is not the first Open Source project with lacking documentation).

And having written all this, I have now created a script that will do all this for us, in unattended fashion. Yes, backups are supposed to be able to run unattended! No, I do not want any manual checking in the midst of a backup process! I do not want to be up at 3AM!

And Yes! I want a way to verify my backups, with some ease! No, I'm not going to set up a cluster of a 8 nodes with 4 Tb disk, just to verify a backup! And this is usually much less of a problem with MySQL, as it is not sharded / distributed. But for anything that IS sharded / distributed, for heavens sake, make sure there are tools to support this. In particular Backup tools!

/Karlsson

MyQuery 3.4.2 available

MyQuery, the ad-hoc query and scripting tool for MySQL, as always running on Windows, supporting servers on any platform, is now available in version 3.4.2. This version has a few minor bug-fixes in the "Break long lines" plugin that cleans up SQL text, and also a few additions to this plugin.
I have not heard of any issues with MyQuery itself since I released 3.4.1, but I will not take the 3.4 version into GA until I have fixed a few more things in the "Break long lines" plugin to turn it into a more full-features SQL beautifier with a lot more features than now.
Until then though, 3.4.2 is what you want, and the Compact Spaces option in the Break Long Lines plugin that is added with this release, should provide useful.

In addition to all this, if you want to meet up and discuss this tool, then I am at your service at the MySQL Usre Conference coming up next week.

Best regards
Anders Karlsson

Using the right tool for the job at hand - MongoDB, MySQL and Sphinx

You may have seen my posting regarding "eventual consistency" some months ago, and you may have come to the conclusion that I was insisting that a SQL based RDBMS is the way to go for just about anything. Tell you what, that is not so. And nether am I against using. say, MongoDB, where it is appropriate.

The whole deal with Eventual consistency is something that I am still opposed to, I want to know if my data is consistent. And I am not not sure that you cannot have a fully consistent, distributed system either. But I guess that debate goes on. And I still want my base data to be consistent. Like in RDBMS-SQL-Foreign-keys-all-over-the-place-and-not-a-bl**dy-bit-lost-in-the-MyISAM-swamp consistent. That is what I want the base data to look like. And if there are compromises with this, which it may well be, then I want to know about those too.

So, having covered that, what am I trying to say? Well, if you properly normalize your data, then the more you normalize and the more strict you are with data quality, the more troublesome management of that data is going to be, and that is something we have to live with I guess. But if you then are to ask some hefty queries on that data, where the data is organized in such a way to make data real consistent, and the queries just want to data, and the query-side of things really doesn't care about normalization at all, how do you deal with that? One way of course may be to replicate to something more query-friendly, possibly a second MySQL-server or possibly even a bunch of such servers in a scale-out scenario. But your data structures still look really complex, having being built to support storage, update, maintenance and consistency requirement foremost.

At Recorded Future we have taken a different path in our latest release: Choose the best tool for the job at hand. We use MySQL with InnoDB for our data loading and storage. And for that, MySQL worka real well. So we have the data we have collected and processed and organized, structed nicely in an RDBMS.

Now, on the other side of things, where queries are made, things look different, there we want to fulfil 2 needs, basically:

• Fast quering for data, in out case these are instances.
  
• Fast retrieval of attributes of the instances that was retrieved.

The above is fulfilled with 2 supporting technologies, Sphinx for instance querying and MongoDB for instance attribute (anwering the typical kind of key-value-store query: Here is a key, gimme all the data). This is what we are live with now, and so far we are really happy. Mind you, this didn't come that easy, both Sphinx and MongoDB has caused some problems.

Forgetting Sphinx for now, what we do with Sphinx is actually really simple, and concentrating on MongoDB, where we also do pretty simple things, but the requirements and the scale in the case of MongoDB is higher for us. MongoDB so far has performed well for us. We are running in an Amazon EC2 environment, and that has issues of it's own (in particular this seemed to be the case with Sphinx, but they are on the case). As for Mongo, this is so common in EC2 environments so I guess ot has been more tested.

I admit to liking MongoDB. It is lightweight, easy to manage and has some cool features, such as automatic, and reasonably transparent, sharding. MongoDB at first seems to have good documentation, there is a lot of examples and samples. What is lacking is a good reference though, and a good administration guide is also missing and the Sharding support is still not well documented from admin point-of-view. The JavaScript orientation in MongoDB at first was annoying to me, but increasingly I find it useful. One thing to note about MongoDB is that it needs it's memory: the more the better. For good performance with MongoDB you really want to avoid disk accesses (in particular in EC2 environments). What we do to avoid disk-accesses as much as we can, we have gzipped the data items. We store data in JSON format (yes, we are a JSON shop, sorry XMLers out there) and compressing data fast and having a fast JSON parser is crucial. And yes, we DID think about the case with frequently accessed data: we have the option, on an instance-by-instance basis, to store data as plain JSON or compressed. Plain JSON has advantages, such as as MongoDB is using JavaScript, JSON is pretty much Native to MongoDB, making certain operations real easy.

We are always on the lookout for new technologies, and we do try many things, but the current setup is really useful and we do get much better performence and scalability. And yes, we do get both, with the same number of servers, we get better performance, and much better distributed load of the machines. Now we are waiting for Amazon to fix their disk IO and Network issues.

Your truly
/Karlsson
Hope to see you in Santa Clara in April at the MySQL UC!

Announcement: MyQuery 3.4.1 Released

MyQuery 4version 3.4.1 is now released on Sourceforge: http://sourceforge.net/projects/myquery/. This is a pretty minor release with just a few fixes, but the first of them pretty significant:

• Windows 7 User Access Control (UAC) issue fixed - When running on Windows 7, the Windows UAC would prohibit MyQuery from starting, complaining about registry access. For once, this was not yet another UAC annoyance, but an actual bug that I hadn't caught before. Now, this should work OK even on Windows 7.
• SQL Statement normalization issues - The SQL Statements dialog got statements wrong at times, and could cause a crash. This was due to a bug in the normalization of SQL Statements and this is now fixed, including refining the normalization a bit.
• Optionally Show SQL SELECT tool output vertically - This will turn the output from the SQL command in a SELECT tool around. This is for running statements that will produce multiple columns instead of multiple rows, such as SHOW SLAVE STATUS and SHOW MASTER STATUS. The output from these commands is weird, the multi-column layout really isn't useful, but this little feature fixes that.

Happy SQL Coding!
/Karlsson

It's about time

TIMESTAMP and DATETIME is how dates and times, or temporal, data is represented ion MySQL. These types are combined with a bunch of operators on these types, such as ADDATE and TIME_FORMAT and such. So is this enough to effectively manage temporal data then? Nope, in my mind, far from it.

To begin with, time isn't such a simple type of data as we might look at it. Is a date just a count of the number of seconds since 00:00:00 on Jan 1 1970 or something like that? And we then convert it to a calendar and all is fine? Is time data really linear in that sense? No, it's not. Dec 24 2010 is Christmas Eve and when you get your Christmas gifts here in Sweden. Is this significant? Also, it's a Friday. That this data is a Friday can be calculated from the date or the seconds-since-jan-1-1970 counter, but that is a different thing. Looking at that counter, you cannot see that there is no way to look at that number and figure out that this day is special, but it is.

To manage this, we have the date and time functions in MySQL, which help us figure things like the above out, but there are issues with it still. To begin woth, the constructs quickly turn difficult. Secondly, you cannot really do an indexed search on temporal data, except pretty simple cases, such as searching for a data and time, before or later than a given data and things like that. A search for "all saturdays" will sure not use an index.

There are more issues though, like that fact that, as I have shown above, a temporal value as we know them above, contains hidden data, something that is considered bad (like the day of the week, the month, the fact that a data is a leap day etc).

And it doesn't end there, there are different calendars in use, in different parts of the world and in different inductries. If you have a mortage on your house, chances are that the interest you pay is the same each month, despite the fact that the interest in calculated over the full year and different months have different number of days. The reason is that interest uses a "360 day" calendar, where each month is assumed to have 30 days. And different financial instruments use diferent calendars (for example the one called 366 day, which is our normal calendar and the 365 day calendar. which is the normal calendar with the exception that there is no leap day). If you try to match one financial instrument to another, when they use different calendars, is not a lot of fun (I've done it, so I know).

But chances are that many of you now think that this is not a big problem for you, as you don't see people asking questions on data with complex temporal relationships. And to that I say that, there is a big chance that this is because:

• Temporal data is difficult. More so than simple integers and strings, disregarding the simple before date X or something like that. But we just don't ask the questions like "how much of the sales per quarter happen on the last day of the quarter" and things like that. Which doesn't mean the information isn't interesting, but the questions is difficult to formalize and even more so to execute in a database. Not to mention checking for overlapping dates and things like that.
• We are used to not being able to ask these kinds of questions.

But really we should be able to ask, query and get data with a lot more temporal information. When was the last time you asked a temporal data query in Google (not that you CAN do this). Like, how has the matching or IBM and Pressrelease changed over the last 10 years, i.e. how many Press releases did IBM post in 2001, in 2002 etc. Yes, the data is there somewhere, querying it is difficult. And using temporal data combined with data you get on the web should allow you to determine more than what you can today (i.e. IBM and Press release matches X number of documents in Google right now). Adding a temporal axis to web data should allow you to determine what might happen.

One issue with all this is that databases are hardly geared up to support this, and a second is that the data to support this isn't available in large quantities yet. This is one of the things we are looking at at Recorded Future, but there is more stuff coming. The web is there with masses of data, now we need to make sense of all that data. One aspect which is there but which hasn't yet been explored much is just the temporal aspect of data.

/Karlsson

A life among databases...

A long time ago, in the early 1980's, I decided to change jobs. I was a young guy, with no real experience of commercial software or anything like that, rather I was a self-taught sysadmin for an ancient UNix system. The company I worked for was in the Telco business, so I looked for another job where I could develop myself and at the same to use my telco knowledge.

I found a Telco startup, privately held. I must say that the fact that it was privately held meant nothing to me at the time. Nothing. They were building a system, the servers were VAXes running VMS, and again a became a sysadmin.

Having been sysadmin at this company for a while, building up the central datacenter (to be honest, in todays words, that was what I was doing, but at the time, I had no real clue. I was mosr enthusiastic and ready to take on any task than I was smart or intelligent or knew what I was doing, really). But I wanted to develop myself, and at the previous job I had learnt to code in C (it was a Unix system after all), so I slowly migrated into database development, managing sysadmin duties on the side.

Still, I wasn't truly professional I think. But I was willing to work and I was persistent and just wouldn't let go. I came to the office dressed in a pair of Jeans and a T-Shirt, and wasn't really aware that sometimes it would be a good idea to dress up or something (this was the 1980's still, so that might have been a good idea back then).

As a developer, I realized that the system used a database, and a SQL database! I had no clue whatsoever what this was. But I started writing code creating tables and working my way through this, learning as I went. That you needed something called an "INDEX" became obvious to me after I had shown my latest creating to my colleagues and the things was just soooo slow. In the end, I actually picked up the manual for that "SQL Database", whatever THAT was.

After about a year at this company, they decided to move their operations abroad, and I wanted to stay in Sweden, so I went looking for another job. The company behind the SQL Database I had used was looking for people it seemed, so I applied for a job. What it was like working for a US based software company was something I had no clue about. I got the job and turned up for my duties as a support engineer dressed in Jeans and a T-Shirt, and got to work. What a support engineer was really supposed to be doing wasn't something I really knew, it was more along the lines of people calling me with questions, and I tried to help them, as best as I could.

The company in question was Oracle. And Oracle really did support me, and courage me, to develop myself, to go to training classes (I didn't ask for these, I was just sent away on them), to take on other jobs inside the organization to to develop my technical and business skills.

For all this, I am grateful. Oracle largely shaped me for my future career among database companies, and if that is a good or bad thing is up for you to decide. Now I'm back at Oracle, and I still enjoy it. I am aware that not everyone will agree with me here, but I am glad to be back, after nearly 20 years after I left, and many things with this geart company is still around.

All in all, I'm sure Oracle is a good home for MySQL. You may think differently, but I am honored to work for Oracle, and even more so with MySQL at Oracle. Frankly, I can't see that it can get any better.

/Karlsson

Running MySQL Cluster as a Service on Windows

The MySQL Cluster daemon for MySQL Cluster (ndbd and ndb_mgmd) doesn't by themselves yet let them run as a service (apparently ndb_mgmd does, but I haven't seen it documented anywhere on how to do that). But there are ways to fix this, using some simple Windows tools and some registry hacking.

What you need to find is the Windows Resource Kit from some version of Windows that includes instsrv.exe and srvany.exe. It is not too picky with the actual version of Windows you run it seems, I used the Windows NT 32-bit versions of these on a 64-bit Windows 7 box, and it works just fine.

These two programs are simple and are easy to use:

• instsrv allows you to install a service, it's real simple, just run the program and it will show the options (and these are few).
• srvany allows you to run any odd program, that is not intended run as a service, do do this anyway.

Now, Google a bit and download these two puppies and open a DOS Window and change directory to where you installed these guys. Now we are realy to install three services (1 for the mgm and two datanodes), so they will run the srvany program:
F:>ResKit> instsrv mgm1 F:\ResKit\srvany.exe
F:>ResKit> instsrv ndbd1 F:\ResKit\srvany.exe
F:>ResKit> instsrv ndbd2 F:\ResKit\srvany.exe

These three services are rather boring now, they do not do anything, really. To fix this, open the registry editor (regedit) and open:
HKEY_LOCAL_MACHINE\System\CurrentControlset\services\mgm1
Now, we need to define what srvany.exe should run. In the key above, create a new key called Parameters. In this key then, create two values of the type REG_SZ:

• Application - The full path to the program srvany should run.
• AppParameters - The parameters to pass to the program srvany will run.

In my case, I set these to, for the mgm service:
Application: F:\MySQL714\bin\ndb_mgmd.exe
AppParams: --config-file=F:\MySQL714\mgm1\config.ini --configdir=F:\MySQL714\mgm1

I think you see where I'm going with this, but just to be sure, this is what I set for the ndbd1 service:
Application: F:\MySQL714\bin\ndbd.exe
AppParams: --ndb_connectstring=nodeid=2;host=localhost:1186

And that's it, folks! Well, one more thing I guess, we want to install the MySQL Server as a serive also, but this knows how to install itself as a service and can also run nicely as a service, so this is easy:
F:\ResKit>F:\MySQL714\bin\mysqld --install MySQL714 --defaults-file=F:\MySQL714\my.ini

Make sure that the ini-file for the MySQL server is configured to access the cluster we just set up, and to have the ndbcluster engine enabled (for some reason, the latter is not enabled by default). Do this by editing the my.ini file used by the server (and referenced above) and add the following two lines to the [mysqld] section:
ndb-connectstring=host=localhost:1186
ndbcluster

Now we have created and configured a bunch of services for a MySQL Cluster setup, and to try it out, lets start them using the NET command line program (the SC command may also be used):
F:\ResKit> net start mgm1
F:\ResKit> net start ndbd1
F:\ResKit> net start ndbd2
F:\ResKit> net start MySQL714

Have fun! And just a word of caution: Although this works just fine for testing, note that shutdown of the ndbd and mgm services isnt really graceful here, so this is just for fun and testing. Also, ndbd and ndb_mgmd will sure be possible to run as a Windows service in their own right, in some future version of MySQL Cluster.

/Karlsson

MySQL Cluster NDB MGM API on Windows

As MySQL Cluster is now available, and GA, on Windows, maybe it's time for some NDB API coding on that platform, right? The reason for this might, as it is in my case, be that Windows is a pretty good GUI Desktop platform, and MySQL Cluster / NDB really needs something like this. Those of you who have followed and used Cluster for a while, might remember my ndbtop tool that I created way back, and which is a MySQL Cluster monitor for Linux using ncurses. This is still useful I guess, but as far as a nice GUI presentation goes, ncurses leaves a lot to be desiered, to say the least.

So where do we start on Windows then? Well, to be honest, MySQL Cluster on Windows doesn't currently come with an installer, it's just a .zip file to unpack. But we are only using NDBAPI and the NDBMGMAPI, so that it no big deal, right? Just unpack the zip-file, write your C or C++ and compile with the supplied includefiles and libraries? Right?

Nope, sorry to disappoint you, but it is more complicated than that. MySQL Cluster on Windows doesn't include ndbapi in the binary release. You might think it does, considering there is a ndbapi.lib file and stuff. But this is not enough, you want the includefiles and also a few more libraries. Also, these are static libraries, which will confuse things a bit. Let me explain this first:

Microsoft has a habit of introducing new things in the C and C++ compilers that rely on things in the C runtime library. This is not a good habit, as it break compatability when building static libraries in particular. What happens is that the static library contains references to the embedded C RTL, and if I link with that library, I will have conflicts between the C RTL that my program wants, and the one that is in the library. And if I link my application without including the standard C RTL, and rely on the one in the library I am linking with, then things may break as my compiled code may contain references that are part of the C RTL I am using, but isn't part of the C RTL of the library I am linking with.

Anyone who has, on Windows, linked with the static MySQL Client library (mysqlclient.lib) knows what I am talking about here, and you also know the solution out of this mess: Compile the library yourself with the Microsoft compiler you use for your application.

So we need to revert to the source anyway. I use Visual Studio 2010 these days, but this is not really supported by the MySQL Build process, or to be honest, I haven't even tried it. I have VS 2008 installed on the same machine, build the project for that, and then let VS 2010 convert it.

So go along and do what is usually done when building MySQL from Source on Windows:

• Get cmake from http://www.cmake.org.
• Use a DOS prompt and go to the main directory where you unpacked the MySQL Cluster sources.
• In the DOS Window run the command:
  win\configure.js WITH_NDBCLUSTER_STORAGE_ENGINE
• In the DOS Window run the command:
  win\build-vs9
• All this will create a MySql.sln file in the main directory of the MySQL Cluster sources.
• Open the MySql.sln file with the Visual Stuio versioon of your choise.

Having done all this, select the Release build and build the following projects:

• mysys (mysys\Release\mysys.lib)
  
• strings (strings\Release\strings.lib)
  
• ndbgeneral (storage\ndb\src\common\util\Release\ndbgeneral.lib)
  
• ndbportlib (storage\ndb\src\common\portlib\Release\ndbportlib.lib)
  
• ndblogger (storage\ndb\src\common\logger\Release\ndblogger.lib)
  
• ndbtrace (storage\ndb\src\common\debugger\Release\ndbtrace.lib)
  
• ndbmgmapi (storage\ndb\src\mgmapi\Release\ndbmgmapi.lib)

You can of course build the whole shebang also, if you so wish. Now, after all this fun, what's the next step? First copy the above libraries (the paths are in prenteses after the library names) to some easy to remember place. I have downloaded the MySQL Cluster Binary also, and have placed these files in a new ndbapi directory under the lib directory. In addition, you need to copy a few includefiles, in my case to the MySQL Cluster binary include directory:

• ndb_init.h (storage\ndb\include\ndb_init.h)
  
• ndb_types.h (storage\ndb\include\ndb_types.h)
• ndb_constants.h (storage\ndb\include\ndb_constants.h)
• the whole mgmapi directory (storage\ndb\include\mgmapi)

The files are copied in my case to the location where I have the MySQL Cluster Binary includefiles, and the mgmapi directory is copied as sibdirectory to this).

Whoa, that was a lot of fun and copying! Right? Or?
Anyway, you should now be able to build a MySQL Cluster NDB MGM API application. Note that we did a Release build of the libaries, so to avoid warnings, build your application in Release build also. Include the path to the MySQL Cluster Binary include directory for the compiler and point the linked to the lib directory created above. And then one more this:
The static libraries already have a reference to the C Run Time library, so tell Visual Studio to disreard the LIBCMT default library. You do this Linker->Input settings in the "Ignore Specific Default Libraries", which you set to LIBCMT. If you build from the commandline, this is the same as /NODEFAULTLIB:"LIBCMT"

Having done all this, your application should compile, link and run just fine. For your reference, here is a Windows command-line program that will show the nodes in the Cluster and the status of them. And yes, the whole idea was to build a GUI application, but it's just too much code for a sample like this. Here you go anyway:

#include
#include
#include
#include
#include
#include
#include

typedef struct tagTEXTMAP
{
int nValue;
char *pText;
} TEXTMAP, *PTEXTMAP;

TEXTMAP g_NodeType[] = {
{ NDB_MGM_NODE_TYPE_API, "API" },
{ NDB_MGM_NODE_TYPE_NDB, "Data" },
{ NDB_MGM_NODE_TYPE_MGM, "MGM" },
{ NDB_MGM_NODE_TYPE_UNKNOWN, "Unknown" },
{ 0, NULL }};

TEXTMAP g_NodeStatus[] = {
{ NDB_MGM_NODE_STATUS_UNKNOWN, "Unknown status" },
{ NDB_MGM_NODE_STATUS_NO_CONTACT, "No contact" },
{ NDB_MGM_NODE_STATUS_NOT_STARTED, "Not started" },
{ NDB_MGM_NODE_STATUS_STARTING, "Starting" },
{ NDB_MGM_NODE_STATUS_STARTED, "Started" },
{ NDB_MGM_NODE_STATUS_SHUTTING_DOWN, "Shutting down" },
{ NDB_MGM_NODE_STATUS_RESTARTING, "Restarting" },
{ NDB_MGM_NODE_STATUS_SINGLEUSER, "Single user" },
{ NDB_MGM_NODE_STATUS_RESUME, "Resuming" },
{ 0, NULL }};

int main(int argc, char *argv[])
{
NdbMgmHandle hMgm;
ndb_mgm_cluster_state *pState;
ndb_mgm_node_state *pNodeState;
PTEXTMAP pMap;
char *pNodeType;
char *pNodeStatus;
int i;

// Check arguments.
if(argc < 2)
{
fprintf(stderr, "Usage: %s \n", argv[0]);
return 1;
}
ndb_init();
hMgm = ndb_mgm_create_handle();

// Connect to mgm server.
ndb_mgm_set_connectstring(hMgm, argv[1]);
if(ndb_mgm_connect(hMgm, 0, 0, 0) == -1)
{
fprintf(stderr, "Error connecting to cluster\n");
return 1;
}

pState = ndb_mgm_get_status(hMgm);
for(i = 0; i < no_of_nodes; i++)
{
pNodeState = &pState->node_states[i];
for(pMap = g_NodeType; pMap->pText != NULL && pMap->nValue != pNodeState->node_type; pMap++)
;
if(pMap->pText == NULL)
pNodeStatus = "Unknown type";
else
pNodeType = pMap->pText;
for(pMap = g_NodeStatus; pMap->pText != NULL && pMap->nValue != pNodeState->node_status; pMap++)
;
if(pMap->pText == NULL)
pNodeStatus = "Unknown status";
else
pNodeStatus = pMap->pText;

printf("Node id: %d at %s. Type: %s. Status: %s\n", pNodeState->node_id, pNodeState->connect_address,
pNodeType, pNodeStatus);
}
}

/Karlsson