Another Word For It Patrick Durusau on Topic Maps and Semantic Diversity

September 11, 2012

…1 Million TPS on $5K Hardware

Filed under: Alchemy Database,Systems Research — Patrick Durusau @ 2:47 am

Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

Got your attention? Good. Read on:

My name is Russell Sullivan, I am the author of AlchemyDB: a highly flexible NoSQL/SQL/DocumentStore/GraphDB-datastore built on top of redis. I have spent the last several years trying to find a way to sanely house multiple datastore-genres under one roof while (almost paradoxically) pushing performance to its limits.

I recently joined the NoSQL company Aerospike (formerly Citrusleaf) with the goal of incrementally grafting AlchemyDB’s flexible data-modeling capabilities onto Aerospike’s high-velocity horizontally-scalable key-value data-fabric. We recently completed a peak-performance TPS optimization project: starting at 200K TPS, pushing to the recent community edition launch at 500K TPS, and finally arriving at our 2012 goal: 1M TPS on $5K hardware.

Getting to one million over-the-wire client-server database-requests per-second on a single machine costing $5K is a balance between trimming overhead on many axes and using a shared nothing architecture to isolate the paths taken by unique requests.

Even if you aren’t building a database server the techniques described in this post might be interesting as they are not database server specific. They could be applied to a ftp server, a static web server, and even to a dynamic web server.
…

My blog falls short of needing that level of TPS per second but your experience may be different. 😉

It is a good read in any case.

February 29, 2012

AlchemyDB – The world’s first integrated GraphDB + RDBMS + KV Store + Document Store

Filed under: Alchemy Database,Graphs,SQL — Patrick Durusau @ 7:20 pm

AlchemyDB – The world’s first integrated GraphDB + RDBMS + KV Store + Document Store by Russell Sullivan.

From the post:

I recently added a fairly feature rich Graph Database to AlchemyDB (called it LuaGraphDB) and it took roughly 10 days to prototype. I implemented the graph traversal logic in Lua (embedded in AlchemyDB) and used AlchemyDB’s RDBMS to index the data. The API for the GraphDB is modeled after the very advanced GraphDB Neo4j. Another recently added functionality in AlchemyDB, a column type that stores a Lua Table (called it LuaTable), led me to mix Lua-function-call-syntax into every part of SQL I could fit it into (effectively tacking on Document-Store functionality to AlchemyDB). Being able to call lua functions from any place in SQL and being able to call lua functions (that can call into the data-store) directly from the client, made building a GraphDB on top of AlchemyDB possible as a library, i.e. it didn’t require any new core functionality. This level of extensibility is unique and I am gonna refer to AlchemyDB as a “Data Platform”. This is the best term I can come up with, I am great at writing cache invalidation algorithms, but I suck at naming things :)

Another graph contender! It’s a fairly long post so get a cup of coffee before you start!

An observation that interests me:

It is worth noting (at some point, why not here:) that as long as you keep requests relatively simple, meaning they dont look at 1000 table-rows or traverse 1000 graph-nodes, your performance will range between 10K-100K TPS on a single core w/ single millisecond latencies, these are the types of numbers people should demand for OLTP.

Are we moving in the direction of databases that present “good enough” performance for particular use cases?

A related question: How would you optimize a graph database for particular recursive graphs?

January 28, 2011

Alchemy Database: A Hybrid Relational-Database/NOSQL-Datastore

Filed under: Alchemy Database,NoSQL — Patrick Durusau @ 7:59 am

Alchemy Database: A Hybrid Relational-Database/NOSQL-Datastore

From the website:

Alchemy Database is a lightweight SQL server that is built on top of the NOSQL datastore redis. It supports redis data-structures and redis commands and supports (de)normalisation of these data structures (lists,sets,hash-tables) to/from SQL tables. Lua is deeply embedded and lua scripts can be run internally on Alchemy’s data objects. Alchemy Database is not only a data storage Swiss Army Knife, it is also blazingly fast and extremely memory efficient.

  • Speed is achieved by being an event driven network server that stores ALL data in RAM and achieves disk persistence by using a spare cpu-core to periodically log data changes (i.e. no threads, no locks, no undo-logs, no disk-seeks, serving data over a network at RAM speed)
  • Storage data structures w/ very low memory overhead and data compression, via algorithms w/ insignificant performance hits, greatly increase the amount of data you can fit in RAM
  • Optimising to the SQL statements most commonly used in OLTP workloads yields a lightweight SQL server designed for low latency at high concurrency (i.e. mindblowing speed).

The Philosophy of Alchemy Database is that RAM is now affordable enough to be able to store ENTIRE OLTP Databases in a single machine’s RAM (e.g. Wikipedia’s DB was 50GB in 2009 and a Dell PowerEdge R415 w/ 64GB RAM costs $4000), as long as the data is made persistent to disk. So Alchemy Database provides a non-blocking event-driven network-I/O-based relational-database, with very little memory overhead, that does the most common OLTP SQL statements amazingly fast and then throws in the NOSQL Data-store redis to create fantastic optimisation possibilities.

Leaving words/phrases like, blazingly fast, amazingly fast, fantastic optimisation, mindblowing speed, to one side, one does wonder how it performs for a topic map?

Reports welcome!

Powered by WordPress