They say: "Relational Databases Aren't Dead"

This is a good read, claiming: "Relational Databases Aren't Dead. Heck, They're Not Even Sleeping", http://readwrite.com/2013/03/26/relational-databases-far-from-dead. A key quote:

"While not comprehensive, the uses for NoSQL databases center around the acquisition of fast-growing data or data that does not easily fit within uniform structures."

There were 2 parts in the statement about NoSQL's uses. I'll start with the latter:


"data that does not easily fit within uniform structures" - NoSQL is probably the right choice, hmm although I always encourage thinking and architecting in advance. And also online structure changes do exist in the RDBMS world and recently in MySQL: http://dev.mysql.com/doc/refman/5.6/en/innodb-online-ddl.html...
I would definitely warn about the caveats of NoSQL when it comes to actually use and query the data that is so easily stored there...

"acquisition of fast-growing data" - is no longer a no-go for RDBMS and MySQL database. Distributed RDBMS solutions do exist today and they can exploit performance and scalability from the good old MySQL itself

What do you think?

Partial partitioning and sharding

I came across this: http://stackoverflow.com/questions/14136633/difference-between-partial-replication-and-sharding

I was wondering if sharding is an alternate name for partial replication or not. What I have figured out that --

• Partial Repl. – each data item has only copies at some but not all of the nodes (‘Sharding’?)

• Pure Partial Repl. – has only copies of a subset of the data item but no node contains a full copy of the database

• Hybrid Partial Repl. – a set of nodes are full replicas and another set of nodes are partial replicas

I thought it was a good topic, I write a really nice answer, but there was a problem when I pressed "post this answer", probably some error or mistake on their side. Anyway - this is what I have to say about partial partitioning and sharding:

Partial replication is an interesting way, in which you distribute the data with replication from a master to slaves, each contains a portion of the data. Eventually you get an array of smaller DBs, read only, each contains a portion of the data. Reads can very well be distributed and parallelized. 

But what about the writes? 

Those are still clogged, in 1 big fat lazy master database, tasks as buffer management, locking, thread locks/semaphores, and recovery tasks - are the real bottleneck of the OLTP, they make writes impossible to scale... As I wrote in many previous posts, for example here: http://database-scalability.blogspot.com/2012/08/scale-up-partitioning-scale-out.html.

Sharding is where every piece of data lives only in one place, within an array of DBs. Each database is the complete owner of the data: data is read from there, data is written to there. This way, reads and writes are distributed and parallelized, real scale-out can be achieved. 

The idea behind sharding is great, the ultimate scaling solution (ask Facebook, Google, Twitter and all the other big guys) but it's a mess to handle, to maintain. It's hard as hell if done by yourself, ScaleBase enables an automatic transparent scale-out machine - that does all that, so you won't have too...

The battle on the OS...

I came across this piece: http://www.zdnet.com/windows-has-fallen-behind-apple-ios-and-google-android-7000008699/, and I'm all nostalgic now...

I had an Atari, my first computer was Apple IIc,  bunch of my friends had a Commodore 64 (I envied them, it was much cooler than the Apple!) and Amiga (wow!!).

Then arrived almost 2 lost decades where people knew* that Wintel is the only thing out there, and couldn't believe there was or ever will be anything else. Just like in the quote from MIB below...

Thank you smartphone, and thank you RIM, Apple and then Google. You made the world a better place.
I know you're not doing anything for us, you're doing it for your own good, to generate value and yield for your stockholders, while pumping ridiculous paychecks to the executives...

Still, you managed to make the world a better, more interesting place to live in, and more challenges to cope with... Big Data, OLTP velocity, app and data sprawl. More challenges for people like me to seek a solution for! You made scalability a big challenge! So, thank you.

BTW, as it turns out to be, I have a Wintel laptop and a Nokia Windows (splendid!) phone, but I'm doing it out of being open minded, and celebrating plurality!


* - The unforgettable quote from "Men In Black":

"...Fifteen hundred years ago everybody knew the Earth was the center of the universe. Five hundred years ago, everybody knew the Earth was flat, and fifteen minutes ago, you knew that humans were alone on this planet. Imagine what you'll know tomorrow"

Database Performance, a Ferrari and a truck

In the last days I got several queries, from colleagues and customers, about one thing I thought it's a given, well well known, but found out differently: "What is database performance?". Is it speed? Is it throughput? What are the metrics and how do you measure?

I tried to refer to an existing link, but then had to write and describe myself. The thing nearest to describing what I think "Database Performance" really is, is this, it's not bad yet I was able to make it even simpler to my esteemed colleagues and more esteemed customers.

Database performance, in an essence, derived from 2 major metrics:
Latency: the time we wait for an operation to finish. Measured in milliseconds (ms) or any other time unit.
Throughput: number of transactions/commands per time unit usually second or minute.

In the classic world of Data Warehouse and Analytics, throughput is usually a non-issue and latency is king. When database grows larger and larger, analytics complex queries take longer and longer to finish, and the demand is "I need speed!".

In the world of OLTP, throughput is the important measure. TPC-C benchmarks for example, measure only throughput (New Order Transactions per Minute). Oracle made it to meet 30,249,688 NO Transactions Per Minute, nice job, we as readers of the results have no way to know if a single transaction tool 1ms and they managed to squeeze thousands of those in parallel in 1 minute to meet this number, or maybe, the scenario transaction took exactly 1 minute, and Oracle managed to perform 30,249,688 such transaction in parallel. The truth is somewhere in the middle, between the 1 millisecond and 1 minute...

In OLTP the latency should be bearable (for some it's 50ms, for some it's 500ms) and stable as throughput must grow and grow as number of users/sites/devices/accounts/profiles grows and grows.

Another key word is predictability. In my OLTP I need predictable good enough, bearable, constant latency performance. I can't afford a 50ms transactions to take 1 minute once every while. I need transactions latency to be some X I can live with, I need it constant and predictable - while throughput is growing.

Not a popular comparison, but very very relevant: A Ferrari and a truck. Both have 500 horsepower.
A Ferrari will take you 200 miles per hour! However a truck will drive a good legal 70, and she'll go same 70 miles per hour with 100 pounds, 1 ton or 20 tons. Constant, stable, predictable. Yea, I'd like to have a Ferrari for my spare time, or to ace a benchmark, but when it comes to backend server infrastructure they're more like a truck to me... and they deliver...

Life's not fair sometimes... at least one of these has definitely got the looks:

"(Cloud) is complete gibberish. It's insane. When is this idiocy going to stop?"

This is Larry Ellison keynotes in Oracle OpenWorld on September 2008. Only 4 years ago.

"The computer industry is the only industry that is more fashion-driven than women's fashion. Maybe I'm an idiot, but I have no idea what anyone is talking about. What is it? It's complete gibberish. It's insane. When is this idiocy going to stop?"

"We'll make cloud computing announcements. I'm not going to fight this thing. But I don't understand what we would do differently in the light of cloud."

The above along with additional marbles are all here: http://news.cnet.com/8301-13953_3-10052188-80.html

Yesterday Larry stood on that same stage and announced Oracle 12c. The c stands for... Cloud!

And what makes Oracle 12c cloud ready?
12c is a "container database." It's function is to hold lots of other databases, keeping their data separate, but allowing them to share underlying hardware resources like memory or file storage. So this way 12c can be used for software-as-a-service tech companies that need a way to let multiple customers access a single database. It's also geared toward large enterprises who may have hundreds of Oracle databases. It would let them consolidate their databases onto less hardware, saving them money on that and making all of those databases easier to manage.

So in short 2 words: multi-tenancy.

Oracle is still Shared-everything, big boxes, and allow virtualization using internal division and allocation of those shared resources to multiple smaller "virtual databases". Indeed, it's great for consolidation and also multi-tenancy.

Cloud? IMHO, to me, cloud is multi-tenancy, but also scale (out) and elasticity. Amazon calls their cloud services EC2, the E stands for Elasticity. The new "DB made for the cloud" has no news about scale-out or about elasticity. Maybe we'll need to wait for Oracle 13e... :)

Also there's a new Exadata database machine, called x3... Yet another bigger box to do all the above. They say it's Oracle competition to SAP HANA.

And finally, we have a new player in the cloud services... Oracle! They'll have a public cloud offering (like Amazon, Rackspace, HP) and also a private cloud, which is a replica of Oracle's public cloud that is put in the customer's own data center. Oracle would still own the hardware and be responsible for running it, securing it and updating it. "as a Service" in your premise. It's interesting and even rhymes...

So it seems someone regained his composure...

Ref: http://www.businessinsider.com/larry-ellison-just-took-on-amazon-with-a-new-cloud-service-2012-9

Being successful like Pinterest without its DB adventures...

I just came across this: "Scaling Pinterest and adventures in database sharding"  (http://gigaom.com/data/scaling-pinterest-and-adventures-in-database-sharding/)

"Pinterest has learned about scaling the way most popular sites do — the architecture works until one day it doesn’t"

Pinterest found out that "the architecture" is not scalable and they turned to development of a Scale Out mechanism also called Sharding.

I find it amazing that sharding, or in other words, the idea of "scale out by splitting and parallelizing data across shared-nothing commodity-hardware" is not supplied "out of the box" by "the architecture" (such as database, load-balancer, any other IT stuff). I'm wondering who was the one that decided that an IT issue like scale-out should be outsourced from the database to the application developers?...

Amazing!!

When was the last time you heard about a PHP or Ruby developer wrote code to enable Scale Out. NEVER! Scale Out in the application layer is enabled easily by a magical box called a load balancer, and you can get one from F5 or wherever for a low 4 digit USD. Commodity! 

But to scale the database? To enjoy the obvious advantages of "scale out by splitting and parallelizing data across shared-nothing commodity-hardware"? - for this the world still thinks developers need to stop investing effort in innovation, better product, competitive business. Instead they need harness their how-databases-really-work skills to write band-aid code to scale the DB. 

Amazing... As you know I took it personally, and have been solving this paradox every day now, by bringing a complete, automatic, out-of-the-box "scale-out machine", that we like to call ScaleBase. I think Pinterest story is great, with a great outcome, but it's not always the case with this complex matter, and a generic, repeatable, IT-level solution for Scale Out can make it much easier for all other "Pinterests" out there to be as successful and make the right choice and enjoy the great benefits - without the tremendous efforts and labor in home-grown sharding.

Facebook makes big data look... big!

Oh I love these things: http://techcrunch.com/2012/08/22/how-big-is-facebooks-data-2-5-billion-pieces-of-content-and-500-terabytes-ingested-every-day/

Every day there are 2.5B content items shares, and 2.7B "Like"s. I care less about GiGo content itself, but metadata, connections, relations are kept transactionally in a relational database. The above 2 use-cases generate 5.2B transactions on the database, and since there are only 86400 seconds a day, we get over 60000 write transactions per second on the database, from these 2 use-cases alone, not to mention all other use-cases, such as new profiles, emails, queries...

And what's the size of new data, on top of all the existing data, that cannot be deleted so easily, (remember why? Get a hint here: http://database-scalability.blogspot.com/2012/08/twitter-and-new-big-data-lifecycle.html). A total 500+TB is added every day, I would exaggerated and assume 98% is pictures and other GiGo content only to leaves us with fuzzy new daily 10TB. There were times Oracle called VLDB to a DB of over 1TB, and here we have 10TB, every day.

So how do FB handle all this? They have a scaled-out grid of several 10000s of MySQL servers.

The size alone is not the entire problem. Enough juice, memory, MPP, columnar - will do the trick.
The but if we put this throughput of 100Ks transactions per second, it'll rip the guts out of any database engine. Remember that it translates every write operation into at least 4 internal operations (table, index(s), undo, log) and also needs to do "buffer management, locking, thread locks/semaphores, and recovery tasks". Can't happen.

The only way to handle such data size and such load is scale-out, divide the big problem to 20000 smaller problems, and this is what FB is doing with their cluster of 10000s of MySQLs.

You're probably thinking "naaa it's not my problem", "hey how many Facebooks are out there?". Take a look and try to put yourself, your organization, on the chart below:

Where are you today? Where will you be in 1 year? In 5 years? Things today go wild and and they go wild faster than ever. Big data is everywhere. New social apps aren't afraid of "what if no one will show up to my party?", rather they're afraid "what if EVERYBODY show up?"

You don't need to be Facebook to need a good solution to scale out your database