Agile DevOps - People and Process then Automation!!!

Use Cloud and DevOps to build infrastructure by rapid agile iterative development using collaborative open source tools:
-Terraform or Cloud Formation for easy orchestration.
-chef cookbooks, puppet modules or ansible playbooks to build servers.
-Test Kitchen to deploy and test them.
-Packer for build server images or containers.
-EC2Dream a graphic user interface that provides a 'single pane of glass' to do agile devops primarily on cloud servers.
-Amazon AWS, Azure, Google Compute Engine, IBM, Openstack, Local and Hosted servers.

Scaling Cloud Databases

To scale applications it has become accepted practice to introduce a load balancer in front application servers and have clients access the applications via the load balancer. With Amazon EC2 it is very simple to hook up your applications to the Load Balancer and Auto Scaling Services. Traditionally scaling SQL databases is expensive and calls for "scaling up" to bigger servers and clusters of servers. Then there are the redundancy and failover requirements to consider!

Currently there are a number of new approaches to scaling databases and in the next few years we will see which approaches become accepted in cloud computing. My guess is that Amazon RDS will introduces improved scaling of the database. Currently, dbShards and Xeround both have solutions running on EC2. Scalebase even works with Amazon RDS so it would be pretty easy to make Scalebase part of the Amazon RDS offering.

Scale-Out SQL Databases
For SQL databases there are a large number of new products to scale out databases horizontally. Many of these are shared nothing databases that use sharding to partition a database.

    ScaleBase - A Database Load Balancer for your existing server (MySQL, Oracle Database, Microsoft SQL Server or IBM DB2).
    Xeround - MySQL Storage Engine Architecture to implement in-memory database utilizing NoSQL concepts.
    dbShards - Application-Aware Sharding.
    TransLattice - more of an distributed application platform, but it contains a distributed relational SQL database that supports eventual consistency.
    NimbusDB - Scalable SQL transactional database.
    VoltDB - in-memory shared-nothing system. Stored procedures become the unit of transaction.
    ScaleDB - Engine for MySQL using shared-disk architecture rather than shared nothing.
    Drizzle - Leaner, faster version of MySQL removing a host of non-core features (including views, triggers, prepared statements, stored procedures, query cache, ACL, and a number of data types).

Sharding is like a load balancer for the data. A big part of sharding is picking a good partitioning scheme.
1. Vertical partitioning. (data related to a specific feature is stored on the same machines)
2. Key-based partitioning.
3. Directory-based partitioning.

NoSQL databases
Lightweight, open source relational databases that do not expose a SQL interface. These are diverse products and can be key value, Document-oriented, or Column-family stores.
    Amazon SimpleDB
and more....
Designed for high availabiliy and eventual consistency rather than guaranteed ACID (atomicity, consistency, isolation, durability) database transactions like SQL databases.

Cache/In Memory Data Grids
A number of commercial and open source products cache Key/Object Stores. Often fronting SQL databases and using the JCache API.
    MemCached. This was often used with MySQL

Given the prevalence of SQL databases it is hard to imagine non-SQL databases becoming the common database approach. More likely "scale out" SQL databases with front-end caches will become the next architecture. Although as application code moves more to Object Relational Mapping (using products like hibernate) NoSQL databases become more viable.

Scaling Databases on the cloud
Are cloud-based memory architectures the next big thing
Distributed Caching Using the JCACHE API and ehcache
The Workday architecture — a new kind of OLTP software stack


Post a Comment