I have uploaded the presentations to the MySQL User Conference website where it says these will be made available, once I have done just that, uploaded them. That didn't seem to happen, at least it seems to take some time, so I decided to do what Guiseppe Maxia did, try out SlideShare, which seems to work just great. Cool!
So my presentations from the MySQL User Conference 2009 are available here:
/Karlsson
Who is finally home again
I am Anders Karlsson, and I have been working in the RDBMS industry for many, possibly too many, years. In this blog, I write about my thoughts on RDBMS technology, happenings and industry, and also on any wild ideas around that I might think up after a few beers.
Tuesday, April 28, 2009
Friday, April 24, 2009
Final thoughts on the MySQL User Conference
I am pretty happy with how the MySQL User Conference developed, considering the announcement earlier in the week. As I have blogged before, I don't mind working for a database company, that's where I have my roots (or root as in -u root) after all. I actually saw very few effects of the announcement during the whole week.
Both talk I did, on libmysqld and on HA with MySQL together with Robert Krzykawski, went very well. The first had a small, but interested, audience, the latter was pretty full with people that also seemed interested. Robert did a stellar job of presenting what bwin games did in practice to achieve High Availability with MySQL, what technologies they looked at and what they eventually ended up with, and how those choices were made and what went well and what didn't turn out so good.
I am pretty tired today, and I will spend most of the weekend traveling back to Sweden, so I deserve a half-day or so off, so I will spend Friday afternoon at the Computer History Museum in Mountain View, which will be fun and interesting. Tomorrow, Saturday, morning I will be a SFO and I will be in Sweden Sunday morning at 7:45.
You think I will relax on Sunday? No way, José! Sunday evening in Stockholm, I am off to my local hangout Akkurat in Stockholm to see the old Swedish progressive music band Dag Vag, and have a few beers. Now we are speaking proper, unpasteurized, unfiltered English ales, and none of that cascade loaded pasteurized stuff I get in the US (which in itself isn't bad, I just have had too much of it for a while).
Cheers
/Karlsson
Who is not allowed (by the SEC) to comment on the fact that he might be an Oracle employee again
Both talk I did, on libmysqld and on HA with MySQL together with Robert Krzykawski, went very well. The first had a small, but interested, audience, the latter was pretty full with people that also seemed interested. Robert did a stellar job of presenting what bwin games did in practice to achieve High Availability with MySQL, what technologies they looked at and what they eventually ended up with, and how those choices were made and what went well and what didn't turn out so good.
I am pretty tired today, and I will spend most of the weekend traveling back to Sweden, so I deserve a half-day or so off, so I will spend Friday afternoon at the Computer History Museum in Mountain View, which will be fun and interesting. Tomorrow, Saturday, morning I will be a SFO and I will be in Sweden Sunday morning at 7:45.
You think I will relax on Sunday? No way, José! Sunday evening in Stockholm, I am off to my local hangout Akkurat in Stockholm to see the old Swedish progressive music band Dag Vag, and have a few beers. Now we are speaking proper, unpasteurized, unfiltered English ales, and none of that cascade loaded pasteurized stuff I get in the US (which in itself isn't bad, I just have had too much of it for a while).
Cheers
/Karlsson
Who is not allowed (by the SEC) to comment on the fact that he might be an Oracle employee again
Thursday, April 23, 2009
MySQL High Availability talk at the MySQL User Conference
At 11:55 today, thursday April 23, me a Robert Krzykawski are doing a talk on real-world High Availability with MySQL in Ballroom F at the MySQL User Conference, and I know I have written on this before, but I do think this could be interesting, not so much because of what I am going to say, but because of what Robert has to add. Roberts has been working with MySQL and availability with it for many years at bWin and will talk about how they ended with the solution they ended up with.
Also, I thought it would be useful to use this blog post as a board for comments and ideas around this session.
See you in a few hours in Ballroom F
/Karlsson
Also, I thought it would be useful to use this blog post as a board for comments and ideas around this session.
See you in a few hours in Ballroom F
/Karlsson
Wednesday, April 22, 2009
MySQL User Conference - Libmysqld talk done, HA coming up
My talk on libmysqld was yesterday, but I am not off the hook yet, as I still have a talk on Thursday on High Availability with MySQL, this time together with Robert Krzykawski of bWin, who will provide a real-world view on High Availability with MySQL. This is going to be a fun, as I think Robert has some interesting experiences when it comes to High Availability with MySQL running on a large website.
I went to Anthony Curtis talk on an External Stored Procedure framework for MySQL, which was exciting, so I will test this a bit more and I already have a few ideas on use of this framework, for example I would want to use this to create library of functions and procedures to augment development of Stored Procedures with MySQL, to aid new development as well as to easy migration.
/Karlsson
Who thinks he works for Oracle today, maybe
I went to Anthony Curtis talk on an External Stored Procedure framework for MySQL, which was exciting, so I will test this a bit more and I already have a few ideas on use of this framework, for example I would want to use this to create library of functions and procedures to augment development of Stored Procedures with MySQL, to aid new development as well as to easy migration.
/Karlsson
Who thinks he works for Oracle today, maybe
Tuesday, April 21, 2009
libmysqld - Embedded announcements today
At the MySQL Users Conference, Ken Jacobs from Oracle announced an embedded version of InnoDB, using a low-level, non-SQL interface, but still providing the transactional and other features of InnoDB. This is an interesting development, and I assume that this could be competitive to libmysqld, which you know is near and dear to my heart.
Frankly, I don't think this is the case. Libmysqld still has the edge on having SQL access and being available with more engines. But I DO see a place for Embedded InniDB which is in the really small devices, where not even libmysqld would fit, but you still need transactional capabilities.
But I do admit that this announcement takes some of the glamour away from my presentation on libmysqld, but I promise that I will keep you posted, I will test Embedded InnoDB and try to figure out where it fits, and how.
So watch this space for more information
/Karlsson
Confused if I am working for MySQL, Sun or Oracle. Or someone else...
Frankly, I don't think this is the case. Libmysqld still has the edge on having SQL access and being available with more engines. But I DO see a place for Embedded InniDB which is in the really small devices, where not even libmysqld would fit, but you still need transactional capabilities.
But I do admit that this announcement takes some of the glamour away from my presentation on libmysqld, but I promise that I will keep you posted, I will test Embedded InnoDB and try to figure out where it fits, and how.
So watch this space for more information
/Karlsson
Confused if I am working for MySQL, Sun or Oracle. Or someone else...
So what will happen at Oracle now?
As I have blogged about before, I have worked for Oracle before, 20 years ago or so. And since then, I have heard a lot of bad bad words on Oracle on the streets, both on Oracle the company and Oracle the database product. Actually, Oracle isn't about any of those things, no, just as in the case on MySQL, it's about people. And good people. You don't run and build a company like Oracle without good people.
To be honest, most I know about the software industry and most of what I know about database technology, I learnt or was allowed to learn when I was with Oracle back then. And I can tell you that backing and supporting me back then wasn't a really obviously good thing to do (I don't know if it is now even). Bur Oracle did, and I owe them one for that.
Also, I like the Oracle product. There are attributes with the product that are different from, say MySQL, and depending on what you look for, Oracle or MySQL might be the better fit. Running Oracle requires a fair bit more hardware and OS resources, but given that, it's a good RDBMS and if you use the features it has, it is generally good software. And this isn't a statement I made up just now, in view of the current event, but this is something I have said many times over the years, and I have said it because I believe that this is the case.
And one more thing, in Montys blog he states that "most of the MySQL leaders (both commercial and project) have left Sun and the people who are left are sitting with their CV and ready to press send." And I have tow things to say about this:
Who will hang around here, and is a bit excited of meeting old Oracle friends again.
To be honest, most I know about the software industry and most of what I know about database technology, I learnt or was allowed to learn when I was with Oracle back then. And I can tell you that backing and supporting me back then wasn't a really obviously good thing to do (I don't know if it is now even). Bur Oracle did, and I owe them one for that.
Also, I like the Oracle product. There are attributes with the product that are different from, say MySQL, and depending on what you look for, Oracle or MySQL might be the better fit. Running Oracle requires a fair bit more hardware and OS resources, but given that, it's a good RDBMS and if you use the features it has, it is generally good software. And this isn't a statement I made up just now, in view of the current event, but this is something I have said many times over the years, and I have said it because I believe that this is the case.
And one more thing, in Montys blog he states that "most of the MySQL leaders (both commercial and project) have left Sun and the people who are left are sitting with their CV and ready to press send." And I have tow things to say about this:
- I don't know what a "MySQL Leader" is.
- I' still around and I am not sitting with my CV.
- I've been with MySQL for 5 years, so I would like to think it has some significance.
- Actually, no one I have spoken to so far are "ready to press send".
Who will hang around here, and is a bit excited of meeting old Oracle friends again.
Monday, April 20, 2009
Virident hardware acceleration for MySQL
Today Virident announced a set of servers, called GreenCloud, aimed at increasing performance for MySQL and memcached servers. Last week I got a chance to talk to Vijay Karamcheti and Shridar Subramanian at Virident about their technology and get a preview on what they are up to.
The technology Virident use to improve performance is a third level of memory storage based on Flash. But it goes way beyond just adding SSD disks. To put things in perspective, look at how resources in an average server has developed in the last 20 years or so. We have now something like 1000 times more memory, and 1000 times more CPU performance, but disk performance has increased very little, maybe 5 times, and that is an optimistic number. Note that this is regarding disk performance, available disk storage has increased also 1000 times or so.
What does this mean then? Well disk I/O is an issue, probably the main issue for database performance. Now, database has still gotten faster, a lot so, as we have more memory and can hence cache A LOT more data, which speeds things up enormously. That performance comes from the fact that we can avoid disk I/O.
There are a couple of issues here though:
But if we go back 20 years in time again, when we were then compensating for slow disks put caching data in RAM, there were compromises being done. Fast, and random, RAM access as opposed to slow disk block-level access. But what has happened now is that there is an even bigger gap in performance between size of RAM and disk performance. So can we not fill that gap?
Looking at attributes of the two types of memory we are looking at so far, in case of RAM:
I want to note that there are other ways of solving this problem. One is to do what MySQL Cluster is doing, which is “semi persisting” RAM by synchronous replication between nodes.
As anyone can realize, applications really need to be aware of this “third storage media” that Virident provides to work properly. Virident has a special version of the InnoDB plug-in to handle this. And the known scalability issues with InnoDB are not really present here either, and least to a much larger extent that in “normal” InnoDB, as this is the InnoDB Plug-in with a lot of fixed for this same problem.
And it doesn’t end there. As I wrote above, for the developer this Flash memory has similar attributes to RAM, i.e. it is not a block-level device but random access, and there are no context switching needed! These are the two features that makes this technology stand away from just plugging in SSD disks in any server!
All in all, I’m excited about this, there is a lot of performance potential to gain from this setup. By being able to scale write-performance on a single server to new higher level, means that technologies, in and of themselves good, like sharding, might be needed asmuch anymore. Also, any distributed technology to solve this problem, like MySQL Cluster, has limitations, cache invalidation and distributed locking, none of which makes for high scalability. Maybe Virident technology will be a standard component in any high-end MySQL server eventually?
The technology Virident use to improve performance is a third level of memory storage based on Flash. But it goes way beyond just adding SSD disks. To put things in perspective, look at how resources in an average server has developed in the last 20 years or so. We have now something like 1000 times more memory, and 1000 times more CPU performance, but disk performance has increased very little, maybe 5 times, and that is an optimistic number. Note that this is regarding disk performance, available disk storage has increased also 1000 times or so.
What does this mean then? Well disk I/O is an issue, probably the main issue for database performance. Now, database has still gotten faster, a lot so, as we have more memory and can hence cache A LOT more data, which speeds things up enormously. That performance comes from the fact that we can avoid disk I/O.
There are a couple of issues here though:
- For writes, I still need to go to the disk, independent of how much RAM I have, a disk I/O will still need to happen, to the database or a logfile, but it must happen. The reason is simple. If I put my written and committed transaction in a log buffer in memory, by transaction will not be persisted.
- Caching of databases only helps so much. Once you have cached up, say, 20 % of the data in the database, further caching will improve performance as much. The reason is of course that data access patterns are skewed, they are not evenly spread across the total size of the database.
But if we go back 20 years in time again, when we were then compensating for slow disks put caching data in RAM, there were compromises being done. Fast, and random, RAM access as opposed to slow disk block-level access. But what has happened now is that there is an even bigger gap in performance between size of RAM and disk performance. So can we not fill that gap?
Looking at attributes of the two types of memory we are looking at so far, in case of RAM:
- Is fast and random accessed.
- But RAM is also not persistent. It is this point that makes disks still so important. Having all the database in RAM is actually possible in many cases these days, but this is not useful, as that data will not be persistent.
- Is persistent and has higher capacity.
- But disks are also slow and use block-level I/O.
I want to note that there are other ways of solving this problem. One is to do what MySQL Cluster is doing, which is “semi persisting” RAM by synchronous replication between nodes.
As anyone can realize, applications really need to be aware of this “third storage media” that Virident provides to work properly. Virident has a special version of the InnoDB plug-in to handle this. And the known scalability issues with InnoDB are not really present here either, and least to a much larger extent that in “normal” InnoDB, as this is the InnoDB Plug-in with a lot of fixed for this same problem.
And it doesn’t end there. As I wrote above, for the developer this Flash memory has similar attributes to RAM, i.e. it is not a block-level device but random access, and there are no context switching needed! These are the two features that makes this technology stand away from just plugging in SSD disks in any server!
All in all, I’m excited about this, there is a lot of performance potential to gain from this setup. By being able to scale write-performance on a single server to new higher level, means that technologies, in and of themselves good, like sharding, might be needed asmuch anymore. Also, any distributed technology to solve this problem, like MySQL Cluster, has limitations, cache invalidation and distributed locking, none of which makes for high scalability. Maybe Virident technology will be a standard component in any high-end MySQL server eventually?
/Karlsson
Back again at Big-O
I was awaken at around 5 this morning, here at the MySQL Users Conference, by a fellow coworker home in Sweden, with the news that Oracle had bought MySQL. This was a bit of a surprise, but also a bit of fun, as I used to work for Oracle, way way back in the 1980's. I was a Sales Engineer most of the time there, just as I am with MySQL, except a Sales Engineer at Oracle is a Pre-Sale enginer (or at least that used to be the case).
This is sure going to be a fun and interesting Users Conference, I can tell you that! And I think I have some of my old Oracle business-cards at home, and also some Oracle T-Shirts and stuff, so I can start being an Oracle dude again. Well, well, that was an interesting turn of events, I'll keep you posted!
/Karlsson
Wondering if my ponytail is fashionable at Oracle...
This is sure going to be a fun and interesting Users Conference, I can tell you that! And I think I have some of my old Oracle business-cards at home, and also some Oracle T-Shirts and stuff, so I can start being an Oracle dude again. Well, well, that was an interesting turn of events, I'll keep you posted!
/Karlsson
Wondering if my ponytail is fashionable at Oracle...
Sunday, April 19, 2009
In Santa Clara now, reaady fo the MySQL User Conference
It's time for the MySQL User Conference now in just a day or so, I have my talks ready, and I been rehearsing them and doing some final editing last week. Except for all the good talks, and meetings with old friends and MySQL expert from inside and outside of Sun, I am also excited to see so many cool things in the Expo.
Most prominent there is the new Storage Engines, like Virident, Kick Fire and InfoBright. Also, we'll see what MySQL has up it's sleves this year (no, I do not know, this will be a surprise for me also). There are many more Storage Engines to look at, including the InnoDB plugin.
A theme here seems to be performance, quite clearly, so we'll see if we have a MySQL Performance shoot-out. It would be cool to benchmark all these software and hardware solutions for MySQL.
Speaking about hardware, I'm also curious to see something more on the Sun Open Storage things. Yes, I know, I work for Sun, but frankly, I know too little about this stuff. The key point of this is, in my mind, the ZFS filesystem, which is WAAAY cool.. Regrettably, ZFS isn't GPL, so we cannot have it on Linux, but I have a campaign to change that.
If you are at the UC, pop by and say hello, if you are not, the continue reading Planet MySQL where most of the blogs from the conference, including mine, are aggregated.
Your just waking up MySQL UC corresondent
/Karlsson
Most prominent there is the new Storage Engines, like Virident, Kick Fire and InfoBright. Also, we'll see what MySQL has up it's sleves this year (no, I do not know, this will be a surprise for me also). There are many more Storage Engines to look at, including the InnoDB plugin.
A theme here seems to be performance, quite clearly, so we'll see if we have a MySQL Performance shoot-out. It would be cool to benchmark all these software and hardware solutions for MySQL.
Speaking about hardware, I'm also curious to see something more on the Sun Open Storage things. Yes, I know, I work for Sun, but frankly, I know too little about this stuff. The key point of this is, in my mind, the ZFS filesystem, which is WAAAY cool.. Regrettably, ZFS isn't GPL, so we cannot have it on Linux, but I have a campaign to change that.
If you are at the UC, pop by and say hello, if you are not, the continue reading Planet MySQL where most of the blogs from the conference, including mine, are aggregated.
Your just waking up MySQL UC corresondent
/Karlsson
Saturday, April 18, 2009
ODBC - Better than you think
"Oh no", you say, "it's ODBC! That horrible, slow and buggy database API! Who in their right mind would use that!"
And I am afraid that I have a slightly different opinion on ODBC than that. I actually think ODBC is pretty good as an API. As far as implementation of the ODBC driver has been for MySQL in particular, well let's be honest and admit that there has been ups and downs, but what is available right now is good and solid, and this has been that case for a while, due to the great team working on MySQL Connector/ODBC right now.
And if you don't know or use ODBC, let me tell you a few facts about it and tell you about some myths:
An interesting aspect of ODBC is that it provides many information functions, like what is the name of th database system (MySQL), what is the version (5.1.32), how are table names and other object names treated? How is sorting handled? Which SQL functions are available. Etc. These are the functions that used in, say, Excel to access almost any RDBMS through ODBC. But note that Excel, to achieve this, only has help from ODBC in the sense that Excel can know how to handle the different RDBMS systems, Excel itself still has to implement it.
I will write more on ODBC later, and I have a command-line ODBC tool that I plan to publish as Open Source eventually. As usual with me, this is rather simple code, but it is useful I think. Right now it is Windows only, although the code should be easy to port to Linux as there is no GUI code.
Happy databasing and see you at the MySQL User Conference!
/Karlsson
And I am afraid that I have a slightly different opinion on ODBC than that. I actually think ODBC is pretty good as an API. As far as implementation of the ODBC driver has been for MySQL in particular, well let's be honest and admit that there has been ups and downs, but what is available right now is good and solid, and this has been that case for a while, due to the great team working on MySQL Connector/ODBC right now.
And if you don't know or use ODBC, let me tell you a few facts about it and tell you about some myths:
- ODBC isn't "slow" or "fast" or "buggy". ODBC is just a specification for an API. The implementation may be slow or fast or buggy or all of those things, but that is a different thing.
- If there is exists a "native" API, ODBC still doesn't have to be slower. For example the native API ight well BE ODBC. Also, an ODBC driver may well work around the native API and make thing faster than the C API that is usually the native one (OK, I know, with MySQL, the API is really the "wire protocol", at least that is the notion in some cases, but the C client API implements that more or less completely).
- Myth: "Using ODBC will mean my application is database independent". This is not true, really. ODBC is a useful API for building database independent applications, but that doesn't mean that ODBC will do the job for you, but it helps you doing the job.
- Myth: "ODBC is one of those horrible Microsoft proprietary technologies". This isn't true either actually. Fact is, Microsoft has been pretty good here. ODBC is an extended implementation of the SAG (SQL Access Group) CLI (Call Level Interface) and ODBC has always been compliant with SAG CLI at some level. (These days, SAG no longer exists as a separate entity, and it became part of X/Open, which is now Open Group).
- Myth: "ODBC is a Windows only technology". This just isn't true, either. Although I have to admit that I had rather see the ODBC layers for Linux etc be based on SAG CLI than on ODBC, but the difference isn't that big actually.
- Myth: "The Driver Manager is the component that is screwing things up". This may well be true, it depends on the Driver Manager you use. But an application can, if it so wishes, ignore the Driver Manager. The reason for this is that Microsoft, when it created ODBC, introduced the Driver Manager, but still wanted to stay aligned with SAG CLI, who said nothing about this actual implementation, it just defined the API. So for Microsoft to align with SAG CLI, but still having an environment where the applications and the drivers were aligned with SAG CLI, despite having three layers, made the DM API the same as the Driver API. There are a few exceptions, mainly that the DM makes things easy for you when it comes to managing different ODBC / SAG CLI versions, so that the DM may map one call that has changed to another call. But excluding the added benefit of the DM, you can link your applications directly against the driver, should you so wish.
- Myth: "ODBC is so GUI oriented with the automatic dialogs and stuff, it's not useful for a server application". This is also wrong, it is not difficult at all to create command-line based ODBC applications, that will not pop up any dialog at all. But then you have to provide any information to ODBC in some other way of course.
An interesting aspect of ODBC is that it provides many information functions, like what is the name of th database system (MySQL), what is the version (5.1.32), how are table names and other object names treated? How is sorting handled? Which SQL functions are available. Etc. These are the functions that used in, say, Excel to access almost any RDBMS through ODBC. But note that Excel, to achieve this, only has help from ODBC in the sense that Excel can know how to handle the different RDBMS systems, Excel itself still has to implement it.
I will write more on ODBC later, and I have a command-line ODBC tool that I plan to publish as Open Source eventually. As usual with me, this is rather simple code, but it is useful I think. Right now it is Windows only, although the code should be easy to port to Linux as there is no GUI code.
Happy databasing and see you at the MySQL User Conference!
/Karlsson
A call for Location Based Services framework
In my mind, Location Based Services (LBS) could well be the next great thing. There are so many things out there that means we are "always on" and there are advantages to that and things that can be done with LBS that this just cannot go unnoticed.
On the other hand, I have found little in the way in frameworks for this, because this is complex, at least if you want it to be useful, and there are many integrations points and the technologies to be integrated are often very different, from end user interaction via maps and by just moving around with a GPS, to back-end services and databases.
Some of you might have read my blog post on a call to action for advancement of the GIS capabilities with MySQL. This (LBS) is exactly why I think this is important. As for any scientific use of GIS, this is of less interest in my mind.
And the reason I think it is important for MySQL is simple: LBS really should integrate with Web 2.0 based services of some kind, that really should be the centerpoint for this. Then linking up to other services and devices, such as Mobile phones, GPSes and WiFi locators and other such things, well that is another issue.
But besides MySQL supporting better GIS we really need an open framework here. I could of no one better to do this than the Open Source community. The framework would, as I see it, have attributes like this:
As can be seen, I do not want to limit myself to only on-line LBS applications, that are closely linked with a GPS usually, but any type of exact or approximate location can be mapped to other objects. And running this in a MySQL GIS enhanced database seems reasonable. And then we add google maps. And CMS frameworks such as Joomla and Drupal could also make good use of this (This can be done today, to an extent, using, say Google Maps, I use it myself on my PapaBlues website, where Google Maps is used to display locations of festivals and places, places that have their geocode-information in MySQL, and I can do even more with this. But I would want this to be even more flexible. And I'm not going to write a GPS integration layer for PapaBlues, so that any GPS could be used to find the closest blues festival, or to download a POI (Point Of Interest) list of bluesclubs to that GPS. But that is exactly what the framework should do. So I admit there is an amount of self-interest here).
/Karlsson
On the other hand, I have found little in the way in frameworks for this, because this is complex, at least if you want it to be useful, and there are many integrations points and the technologies to be integrated are often very different, from end user interaction via maps and by just moving around with a GPS, to back-end services and databases.
Some of you might have read my blog post on a call to action for advancement of the GIS capabilities with MySQL. This (LBS) is exactly why I think this is important. As for any scientific use of GIS, this is of less interest in my mind.
And the reason I think it is important for MySQL is simple: LBS really should integrate with Web 2.0 based services of some kind, that really should be the centerpoint for this. Then linking up to other services and devices, such as Mobile phones, GPSes and WiFi locators and other such things, well that is another issue.
But besides MySQL supporting better GIS we really need an open framework here. I could of no one better to do this than the Open Source community. The framework would, as I see it, have attributes like this:
- Ability to link of with devices are services on both the Client and Server side.
- An ability for the individual user to opt-in and opt-out of any services.
- A flexible Web interface that is accessible and usable to all.
- A means of interacting with the service through an API even on the client side, i.e. if someone wanted to create a completely different looking web interface, they could.
- A means of services to interact of the service level of this.
As can be seen, I do not want to limit myself to only on-line LBS applications, that are closely linked with a GPS usually, but any type of exact or approximate location can be mapped to other objects. And running this in a MySQL GIS enhanced database seems reasonable. And then we add google maps. And CMS frameworks such as Joomla and Drupal could also make good use of this (This can be done today, to an extent, using, say Google Maps, I use it myself on my PapaBlues website, where Google Maps is used to display locations of festivals and places, places that have their geocode-information in MySQL, and I can do even more with this. But I would want this to be even more flexible. And I'm not going to write a GPS integration layer for PapaBlues, so that any GPS could be used to find the closest blues festival, or to download a POI (Point Of Interest) list of bluesclubs to that GPS. But that is exactly what the framework should do. So I admit there is an amount of self-interest here).
/Karlsson
User Conference: libmysqld and Real-world HA
I am now in San Francisco, tomorrow I'll be leaving town for Santa Clara and the MySQL User conference. If you are at the conference, say hello and if there is anything you want to talk to me about, do so, I'll spend my time out and about at the conference.
I am involved in 2 talks:
Traveling to CA this year brought a pleasant surprise for me. I usually fly SAS coming here, and my preferred route is through Chicago. I have been on this particular flight many times. And although I think SAS is an OK carrier, they have been behind some of the others in one aspects: beer. Bad beer is an old story with SAS, and anyone who has been on these flight with me knows that I usually point this out, and I have also emailed SAS on this important subject.
And lo and behold, someone actually listened! This time, they had a reasonably pleasant Danish high-quality lager available from a small brewery!
So all in all, things are looking bright. The expo at the show looks like it will have some good exhibitors with some interesting products. If you really REALLY want to talk to me, ask around at the MySQL booth in the expo, or check in the bar for a pony tailed geek with beer in hand. Chances are it is me.
Happy databaseing
/Karlsson
I am involved in 2 talks:
- On Tuesday at 11:55 I am doing an introduction to libmysqld. If have not used libmysqld, and are interested in what it is and how it can be used, pop by this talk. I will also demo a small libmysqld application, so you know what can be done. It's actually pretty cool.
- On Thursday at 11:55, me and Robert Krzykawski from bWin are to present some real-world experiences with HA for MySQL. I will do an introductory talk here, with some HA for MySQL basics, and then Robert will tell you about the real world. bWin games is a big user of MySQL and has quite a load on their systems, so Robert knows what he is talking about!
Traveling to CA this year brought a pleasant surprise for me. I usually fly SAS coming here, and my preferred route is through Chicago. I have been on this particular flight many times. And although I think SAS is an OK carrier, they have been behind some of the others in one aspects: beer. Bad beer is an old story with SAS, and anyone who has been on these flight with me knows that I usually point this out, and I have also emailed SAS on this important subject.
And lo and behold, someone actually listened! This time, they had a reasonably pleasant Danish high-quality lager available from a small brewery!
So all in all, things are looking bright. The expo at the show looks like it will have some good exhibitors with some interesting products. If you really REALLY want to talk to me, ask around at the MySQL booth in the expo, or check in the bar for a pony tailed geek with beer in hand. Chances are it is me.
Happy databaseing
/Karlsson
Friday, April 10, 2009
Don't forget the spatial extensions
MySQL does have spatial extensions, and I plan to do some more writing on these in the near future. This is the first blogpost on the subject. The spatial extensions in MySQL looks like what they do in most other databases in terms of overall functionality, but the current implementation leaves a fair bit to be desired when it comes to features and implementation. Spatial extensions include extensions in a bunch of areas:
And as if this wasn't enough, only MyISAM supports R-Tree indexing, and not all Storage Engines supports the Spatial (GIS) datatypes.
Thirdly, MySQL can only work with a flat or euclidic coordinate system. This is a disadvantage and will introduce an error, in particular when used with larger geometries, as the world really isn't flat. At least that wasn't the case last time I looked.
And the drawbacks I mention above are just the most basic ones, there are a few more minor omissions and issues.
So are we doomed now when it comes to GIS functions with MySQL? Nope, there has been some work done to fix this. On MySQL Forge you can find a GIS Functions document, about a fix to these issues. This has also been implemented in a variation of MySQL 5.1.23 that is available for download. In addition, the is a talk at the MySQL User Conference on this subject.
So there is hope, and I have a plea for you: Support me in having the 5.1.23 implementation I mention above become part of standard MySQL. There is no reason not to do this, in my mind, but for some reason, it's not happening. Also, download this code, I'd be really happy if someone would want to use this code and develop it further. So yes, there is hope!
And I'll do some more blogging on this subject, if nothing else, I will discuss ways around the current limitations of Spatial Support in MySQL.
/Karlsson
Still opinionated
- Spatial databases. These are datatypes such as point, polygon and geometry etc.
- Spatinal functions. These are functions like distance, size and also functions that check for relations between spatial objects, such as contains and overlaps.
- Spatial indexing. This is typically R-Tree indexes (R for Rectangle) that can be applied when using the Spatial functions above.
And as if this wasn't enough, only MyISAM supports R-Tree indexing, and not all Storage Engines supports the Spatial (GIS) datatypes.
Thirdly, MySQL can only work with a flat or euclidic coordinate system. This is a disadvantage and will introduce an error, in particular when used with larger geometries, as the world really isn't flat. At least that wasn't the case last time I looked.
And the drawbacks I mention above are just the most basic ones, there are a few more minor omissions and issues.
So are we doomed now when it comes to GIS functions with MySQL? Nope, there has been some work done to fix this. On MySQL Forge you can find a GIS Functions document, about a fix to these issues. This has also been implemented in a variation of MySQL 5.1.23 that is available for download. In addition, the is a talk at the MySQL User Conference on this subject.
So there is hope, and I have a plea for you: Support me in having the 5.1.23 implementation I mention above become part of standard MySQL. There is no reason not to do this, in my mind, but for some reason, it's not happening. Also, download this code, I'd be really happy if someone would want to use this code and develop it further. So yes, there is hope!
And I'll do some more blogging on this subject, if nothing else, I will discuss ways around the current limitations of Spatial Support in MySQL.
/Karlsson
Still opinionated
Monday, April 6, 2009
What wrong with SQL_MODE?
Let me tell you what I think is wrong, oh so wrong, with SQL_MODE. And there has been numerous debates and ideas around this for a long time, and many comments on how this really should work. So I assume that you are now all waiting for my very opinionated opinion.
And the problem is that we (the MySQL Server that is) allows it to be set just about everywhere, it's dynamic. This is do NOT like. not one bit. And let me tell you why.
If I work in a small project, like when I'm playing around with my own websites or my own small projects, I am both the DBA and the developer. So there isn't much argumentation going on, I know what I am about to do. But still, I want the database to determine the consistency of data, and the application to be just that: an application that conforms to the rules I have set up in the database schema.
This is similar to my arguments for FOREIGN KEYs. In a small project, I know my application needs to do this or that to keep the internal consistency correct, but the larger the application gets and the more people involved, so with database work, and some with application work maybe in completely different groups, then we have to have a way to set the rules.
Another reason is that setting rules for data in a declarative fashion is so much more foolproof than doing it in the application. If we assume that you have done all the FK stuff in program code, and you want to know what they are. And you didn't write the application, what do you do? If you are using proper FK relationships, CHECK and UNIQUE constraints, just run, say MySQL Workbench, on your schema, and you know what data looks like.
If I make a mistake in teh application in this sense, I may stand to loose data here, as links between objects are "lost". No, even in my small websites (my main site is PapaBlues use InnoDB and foreign keys all over. It's so nice to know that InnoDB will keep me on track here, even if I happen to forget something in some application. Fact is, it is extremely useful and I would have lost data, even in THIS rather simple application (30ish tables).
So, back to SQL_MODE, what's wrong with it? Problem is you can set it anywhere. The same goes for SET FOREIGN_KEY_CHECKS. In a production environment, I want to be able not only to set these, but to enforce them, so some developer, not by spite, but by not knowing better, sets SQL_MODE to something less strict than I want it to be, and hence will corrupt my data.
So then, what about any existing MySQL applications, am I not forgetting about them here? Nope, I'm not. What I think should work here is an extra parameter that allows me to force FK CHECKS and SQL_MODE, if I so wish. If I don't set these, FK CHECKS and SQL_MODE would work just as before. The forcing should be set independently for SET FK CHECKS and SQL_MODE, and have three different values:
Another thing is that setting SQL_MODE in some cases will be tied to an object created with a particular SQL_MODE. I can only image the kind of issues this can cause. The solution: The ability to enforce even existing objects to use a particular SQL_MODE. The way enforcement should, in my work, work then, would be in one of three ways:
And then, when we have that, can we have another thing? A transaction safe data dictionary? Please!
Yours truly opinionated
/Karlsson
I will have a look at testing a forced SQL_MODE eventually
And the problem is that we (the MySQL Server that is) allows it to be set just about everywhere, it's dynamic. This is do NOT like. not one bit. And let me tell you why.
If I work in a small project, like when I'm playing around with my own websites or my own small projects, I am both the DBA and the developer. So there isn't much argumentation going on, I know what I am about to do. But still, I want the database to determine the consistency of data, and the application to be just that: an application that conforms to the rules I have set up in the database schema.
This is similar to my arguments for FOREIGN KEYs. In a small project, I know my application needs to do this or that to keep the internal consistency correct, but the larger the application gets and the more people involved, so with database work, and some with application work maybe in completely different groups, then we have to have a way to set the rules.
Another reason is that setting rules for data in a declarative fashion is so much more foolproof than doing it in the application. If we assume that you have done all the FK stuff in program code, and you want to know what they are. And you didn't write the application, what do you do? If you are using proper FK relationships, CHECK and UNIQUE constraints, just run, say MySQL Workbench, on your schema, and you know what data looks like.
If I make a mistake in teh application in this sense, I may stand to loose data here, as links between objects are "lost". No, even in my small websites (my main site is PapaBlues use InnoDB and foreign keys all over. It's so nice to know that InnoDB will keep me on track here, even if I happen to forget something in some application. Fact is, it is extremely useful and I would have lost data, even in THIS rather simple application (30ish tables).
So, back to SQL_MODE, what's wrong with it? Problem is you can set it anywhere. The same goes for SET FOREIGN_KEY_CHECKS. In a production environment, I want to be able not only to set these, but to enforce them, so some developer, not by spite, but by not knowing better, sets SQL_MODE to something less strict than I want it to be, and hence will corrupt my data.
So then, what about any existing MySQL applications, am I not forgetting about them here? Nope, I'm not. What I think should work here is an extra parameter that allows me to force FK CHECKS and SQL_MODE, if I so wish. If I don't set these, FK CHECKS and SQL_MODE would work just as before. The forcing should be set independently for SET FK CHECKS and SQL_MODE, and have three different values:
- Not enforced - Just like today.
- Enforced with error - Override enforcement will cause an error.
- Enforced with no error - Overriding the enforcement will be silently ignored.
Another thing is that setting SQL_MODE in some cases will be tied to an object created with a particular SQL_MODE. I can only image the kind of issues this can cause. The solution: The ability to enforce even existing objects to use a particular SQL_MODE. The way enforcement should, in my work, work then, would be in one of three ways:
- The server will ignore all this, and work like today.
- The server will refuse to start if there are objects that are not aligned with the server defined SQL_MODE.
- The server will start, but log all offending object states to the error log, and will disable those particular objects.
And then, when we have that, can we have another thing? A transaction safe data dictionary? Please!
Yours truly opinionated
/Karlsson
I will have a look at testing a forced SQL_MODE eventually
libmysqld demo application
This is not a long blogg posting, I just want to point to the simple libmysqld demo application for Windows that I have created. It's available on sourceforge as as I said, this is Windows only so far. I might well go ahead and do a Linux version of it eventually, but right now, this i sfor windows.
But I don't think that matters much, this is not meant to be a terribly useful application in and of itself, rather, this is a minimal application built to demonstrate a concept.
If you are interested in libmysqld, then have a look at the webinar I did couple of weeks, and if you are at the MySQL User Conference, then pop by my talk on the same subject.
Cheers from the database dude
/Karlsson
But I don't think that matters much, this is not meant to be a terribly useful application in and of itself, rather, this is a minimal application built to demonstrate a concept.
If you are interested in libmysqld, then have a look at the webinar I did couple of weeks, and if you are at the MySQL User Conference, then pop by my talk on the same subject.
Cheers from the database dude
/Karlsson
Subscribe to:
Posts (Atom)