Deprecated: Function split() is deprecated in /home/tbriggs/public_html/s9y/plugins/serendipity_event_metadesc/serendipity_event_metadesc.php on line 101

RAM Storage Is Not The Answer

Or, Missing the Point 101

A colleague recently ran some tests with Oracle using only RAM storage in order to improve performance of a DSS application. If I/O is the bottleneck for reporting apps, then why not make the I/O infinitely fast, right? Who cares if there's a hard limit on how much RAM you can put in a box, or if all the data will be lost when the system is shut down?! Heck, this idea can't really be too crazy - there's a whole book on the subject by well-respected authors, after all. And there is at least one company making their business providing large-scale RAM storage with an eye toward database performance, for that matter, so it's not like this is the first time anyone's tried this.

My incredulity aside, the results of the tests were as first surprising but ultimately what I should have expected. Bottom line: in this particular experiment, storing everything (data, temp, undo, rollback, etc.) in RAM provided only a 5-6x performance improvement.

5-6x improvement is nothing to sneeze at, mind you, but it isn't the 10-100x improvement that Netezza has been claiming for years. Nor is it close to what I've seen from Vertica. But how can that be? Isn't I/O the bottleneck, and RAM infinitely fast?

Well, yes, on both accounts. But faster I/O doesn't solve the real problem when running DSS queries against large datasets, which is simply the amount of data that must be touched. Traditional, row-oriented systems must move and manipulate a lot of data in order to satisfy a typical DSS query. Even with infinitely fast I/O, therefore, they must still do an awful lot of work in order to satisfy that query.

In contrast, the new breed of databases - e.g. Netezza, Vertica, etc. - have been designed for reporting and analytics against large datasets. These new systems provide impressive performance improvements over older, transaction-oriented systems for one simple reason: they minimize data movement and manipulation. As a result, for DSS-type queries, they will always be able to outperform Oracle and other traditional DBMSs, even when I/O is infinitely fast.

Netezza and Vertica make great examples of this because they both do it but they go about it in very different ways. Netezza uses a minimalist data access approach that allows large datasets to be processed in small pieces with the least possible effort spent on each piece. Vertica uses a compressed column store that allows aggregate queries to be satisfied with mind-bogglingly small amounts of I/O. In both cases, the key is storing and/or accessing the least amount of data required to satisfy the query, and in a manner that scales as data size grows.

In short: RAM storage is not the answer. Moving and manipulating a lot of data really fast still requires moving and manipulating a lot of data. That will never be as fast as people want or need it to be. As a result, traditional databases will never satisfy reporting and analytics needs for large datasets as well as the newer systems designed for that purpose. Not even if I/O is free.


Trackback specific URI for this entry

One of the responses I received about my comments on RAM storage included this gem: I'm being critical, because I'm an engineer, not because I enjoy being difficult. Speaking as an engineer... amen brother. Preach on.
A helpful reader had this to say in response to my inquiry about EXASOL: The Exasol DWH is a parallel RDBMS running on a cluster of Intel server nodes. The data is stored in shared-nothing manner but with redundant copies on other nodes, so no data is
There's an interesting article over at MySQL Performance Blog that talks about some MySQL benchmarks using RAM-based storage. Here's the most interesting part (to me, anyway): However even with MyISAM we got CPU bound before we could reach the system
Great short and sweet article that makes the point crystal clear about the need for at least two types of databases within an information architecture. One online (for OLTP), and one offline (for Reporting & Analysis). It's clear that one RDBMS cannot do both without compromising.... I hope Microsoft and Oracle are listening to common sense!
I think you may have overlooked the important half of the equation for I/O. Using RAM storage makes the access times infinitely fast, however, it does not make the I/O channel bandwidth infinitely wide. It is quite likely that I/O latency was not the primary bottleneck in your tests, which is why you only saw 5-6x performance gains. It's also likely that the RAM storage and the disk storage had the same connectivity to the host. More details on the test would be beneficial to supporting your claims.
You may have a point - I'll confess to being mostly hardware ignorant. In this test, the RAM used was in the box, i.e. plugged into the motherboard. I wouldn't expect that to fit the "had the same connectivity to the host" criteria you mentioned, but I dunno for sure.
I think there are a variety of issues which could cause the problems you observed.

A modern 2 socket server system can have up to 8 HDDs, which can provide at most 800MB/s of I/O bandwidth, at pretty poor latencies.

That same server can host around 128GB of memory and provide around 20GB/s of bandwidth at around 100-200ns latencies.

I can not speak to what sort of system you were using, and how it was configured, but it makes a pretty big difference.

Given that the difference in peak I/O bandwidths for these two storage media is >20X, there are probably other issues at hand.

First of all, I don't know what sort of DBMS you are using, but there are many techniques which are used for DBMS to get high performance on disks, which are not necessary and probably detrimental for performance on an in-memory configuration. In other words, I have no reason to believe that the comparison was truly apples to apples.

Second, I suspect the reason you aren't seeing a big speed up is that there is another bottleneck lurking in your system. My top candidate would be the processors, or if it was an Intel system, possibly communication between processors. Heck, it could be the network, but I don't know.
Sorry for the late post. I only saw this after Intex in Needham clicked on the link to our site from yours today.

Your comments ignore a great detail of the logic in using RAM for database acceleration.

Server hardware, specifically PCI bridges and IOP capabilities, play one of the most important roles in determining how successful using memory appliances will be. Even though a majority of chipsets can support PCIe payload sizes of 1024, its very hard to find a manufacturer with a payload greater than 256 or in the case of Dell 128. You get what you pay for. PCIe is currently the fastest method of communicating to a server.

The next item is confusion of whether an application is IOP bound or CPU bound. Around 75% of our trials think they are CPU bound when they are really IOP bound. We are able to show that a Dell 2950 can beat the pants off a Dell 6850 when the applicationis IOP bound.

To see the most impact, Async IO or requests in a parallel fashion are suggested. A single threaded app is generally a waste of time, but the number of people that have dual or quad socket machines for this type of application is surprising.

There are also many legacy issues involving IOP limitations that may surprise people. These can be with the Application, iSCSI stack, or other items. We see this impact when doing an iostat while the RAM drive is being written to.

The performance gains we have helped people with have been anywhere from 5x to 300x. The latter has been in cases where there has been the flexibility to tune the application for specific RAM usage.

BTW, for MYSQL Peter's blog was an eye opener that much of the logic with MyISAM and Innodb needs updated to account for RAM as storage. This is similar to users trying to use an appliance for OS swap. The algorithms never allowed for the possibility of having storage at a similar speed as memory. These induce a penalty for writing to a drive. At least in the case of Innodb, the community and Oracle are working hard to address the issue.
Add Comment

Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.
Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.