Service Discovery Solutions In Distributed Systems

Today I participated in a conference in Moscow/Russia devoted to new technologies in the software world.
For those who attended, thanks for your questions! I’m attaching here the presentation:


Toughest backtracking problems in algorithmic competitions



In algorithmic competitions there are frequently problems that can be attacked with recursive backtracking algorithms – a well-known approach to traverse a search tree. Usually, it is good smell, if there’s a goal to analyze all existing combinations of a problem. And, of course, there needs to be the right strategy to meet time limits (e.g. prune it). Here, I’ve decided to talk about a few very interesting backtracking problems I came across. I touch on a backtracking approach to develop in competitors, but bear in mind that, not trying to solve a problem by yourself, seeing the answer up front is a waste of time. Furthermore, this is an advanced level, if you haven’t practiced a recursive backtracking or DFS, please spend some time on basic backtracking problems and come back.

1. Chess Puzzler


This is quite an interesting problem I’ve ever come across, solving it you realize some very important uses cases to consider like memory limits, recursion, combinatorics and optimization techniques. I’ve seen a chess problem in Skiena’s  algorithmic book some time ago, but as turned out, this one is very different.

Problem Statement:

The problem is to find all distinct layouts of a set of normal chess pieces on a chess board with dimensions MxN where none of the pieces is in a position to take any of the others. Assume the color of the piece does not matter, and that there are no pawns among the pieces.

Write a program which takes as input:

•The dimensions of the board: M, N.

•The number of pieces of each type (King, Queen, Bishop, Rook and Knight) to try and place on the board.

As output, the program should yield the number of distinct layouts for which all of the pieces can be placed on the board without threatening each other.


We represent each piece as: “K” – King “N” – Knight “Q” – Queen “R” – Rook “B” – Bishop M – Horizontal size of the board N – Vertical size of the board S – is a set of remaining pieces. For example: Input: 3×3 board containing 2 Kings and 1 Rook, that is S = [K,K,R]. Answer: 4 layouts.


Since we need to find all possible layouts of a chessboard, it can be solved with a recursive backtracking as follows. We take the next piece from S and calculate for it all possible freeSquares on the chess board. Next, by iterating in a loop over freeSquares for current piece, we try to put it in all possible freeSquares. Each loop-step is a potential solution (layout) calls itself recursively by trying to put the next piece for current chess board and so forth until there are no pieces left or freeSquares is empty. Once a piece is placed on the board, we update the set of the free squares by subtracting a set of squares threatened by this piece. In case the set of free squares is empty and there are still any remaining pieces not on the board, there’s no solution to this combination and the recursive function backtracks to the upper level in the recursive tree trying the next loop-step. Thereby, we loop over all steps and stop traversing by pruning impossible configuration in advance – as simple as this. There could be some arithmetic optimization with a number of threatened squares for each piece type by taking into account all remaining pieces to be put on the board and number of free squares, calculated in one go. Since the time limit in this problem was 20 mins to solve, I ditched an optimization. Undoubtedly, my solution can be drastically improved by cutting the search tree even more, and hence I leave this to the reader. Moreover you might want to parallelize this recursive task.

Finishing touch, namely what to do about duplicated pieces like 3 queens or 2 knights etc. Honestly, I spent a great deal of time on this while solving. The thing is that, duplicates are interchangeable in terms of different combinations on the chessboard. For instance, for a board of 1×2 length with free squares [x:1,y:1][x:1,y:2], 2 queens can be placed as [Q1][Q2] or [Q2][Q1] yielding 2 different combinations. A simple solution is to put at once all pieces of one type inside a loop-step. From combinatorics, we can enumerate all C(n, k) unique combinations (aka n choose k) in a single loop. Because we recurse, I created a utility function wrapped around with a standard java iterator which doesn’t have to calculate all combinations up front, rather it traverses them lazily by calculating the next one on the fly. The reason for this was a memory taken on each level of the recursion stack to keep an array of all combinations. E.g. C(n, k) = C(1000,5) results into 8,250,291,250,200 elements. There were also some minor issues with Groovy not being able to correctly calculate a difference between 2 lists of coordinate-pairs. Thanks to guys on stackoverflow who quickly replied with a workaround. The full working code  is now available on GitHub. If somebody of you have an idea to optimize it somehow, please comment one at the end of this post!

2. To backtrack or not, that’s the question: Meet “Mine Sweeper Master” from Google code jam 2014


A tricky and simple at the same time problem was posed last year on Google Code Jam in qualification round – a famous Mine Sweeper master. Yes, the one that comes with Windows operating system – I bet, most of you are aware of! It’s well-known solving minesweeper is NP-complete. But conditions of the problem don’t require you to do that (Please read a problem statement before proceeding).

Solving it with a backtracking is the wrong way, as you are not required to analyze all configurations. The catch is that any correct result is a solution (read carefully a problem  statement)! And thus, you don’t have to attack it with backtracking as this pattern is quite costly, aimed at getting all possible solutions. It is possible, but you won’t pass the large set most likely. Hence, the simplest idea is to start at (0,0) – upper-left corner and fill an area of N cells  with non-mine space from left to right and top to bottom – line-by-line. Further, fill the rest with mines. Clicking the (0,0) cell should reveal if this is a good solution. If (0,0) is not a mine – we have won. If the square contains a 0, repeat this recursively for all the surrounding squares.

There are also a number of important corner cases to consider for this approach:

Single non-mine

If N=1, any configuration is a correct solution.

Single row or single column

If R=1, simply fill in the N non-mines from left-to-right. If C=1, fill N rows with a (single) non-mine.

Too few non-mines

If N is even, it must be >= 4.
If N is odd, it must be >= 9. Also, R and C must be >= 3.
Otherwise there's no solution.

Can’t fill first two rows

If N is even and you can't fill at least two rows with non-mines, then fill the first two rows with N / 2 non-mines.
If N is odd and you can't fill at least two rows with non-mines and a third row with 3 non-mines, then fill the first two rows with (N - 3) / 2 non-mines and the third row with 3 non-mines.

Single non-mine in the last row

If N % C = 1, move the final non-mine from the last full row to the next row.

I was lazy to depict each one. As can be seen, there is bunch of special cases to consider to make this solution pass.

3. Another Chess Board Puzzler: “King” from Google Code Jam 2008

This is one of the toughest problems from Google Code Jam. It differs in that no one solved it in global Code Jam rounds during the round in which it was posed. Algorithmic competitions is like sports, if you feel you can solve easier problems faster – go for it. Otherwise you’re at risk of loosing the competition. Some day next time I will try to attack it too, and for now I say goodbye to  all of you.

Dockerizing Spray HTTP Server


This is the continuation of the previous article. This series shows how simple it is to create a lightweight HTTP-server based on Spray framework, put it into a Docker-image and run multiple instances on any single machine requiring no dependencies.

Implementing a lightweight RESTful HTTP Service

The whole project can be found on GitHub. For impatient, git pull it and jump right to the next section. We’re going to use Scala and Akka framework along with SBT build tool. From Spray framework we will use a spray-routing module which has a simple routing DSL for elegantly defining RESTful web services and it works on top of a spray-can HTTP Server.

Ok, let’s get started.

import spray.routing._
import akka.util.Timeout
import scala.concurrent.duration._
import scala.util.{Failure, Success}

object HttpServer extends App with SimpleRoutingApp {

  implicit val actorSystem = ActorSystem()
  implicit val timeout = Timeout(1.second)
  import actorSystem.dispatcher

  startServer(interface = "localhost", port = 8080) {

    // GET /welcome --> "Welcome!" response
    get {
      path("welcome") {
        complete {
            <p><a href="/terminate?method=post">Stop the server</a></p>
    } ~
      // POST /terminate --> "The server is stopping" response
      (post | parameter('method ! "post")) {
        path("terminate") {
          complete {
            "The server is stopping"
    .onComplete {
    case Success(b) =>
      println("Successfully started")
    case Failure(ex) =>

The REST API is as follows:

  • GET/welcome –> responds with a “Welcome” and Post-hyperlink.
  • POST/terminate –> will stop the server.

DSL describing this API is inside of a method startServer.  And that’s it!

I didn’t want to show the full power of Spray because this article is solely about Docker.

Let’s run it and check:

curl http://localhost:8080/welcome

Dockerizing the server

Because the Docker Engine uses Linux-specific kernel features, I’m going to use lightweight virtual machine to run it on OS X. If you too, just download it, install and run – easy peasy. Just make sure before dockerizing that you’ve set the following 3 env variables for connecting your client to the VM:

export DOCKER_HOST=tcp://
export DOCKER_CERT_PATH=/Users/ivan/.boot2docker/certs/boot2docker-vm

The remaining stuff doesn’t differ much.

We use a trusted automated java build (OpenJDK Java 7 JRE Dockerfile) as a base of our image.

First, you will need to create a Dockerfile for the image in your project:

# Our base image
FROM dockerfile/java


USER daemon

# Here the stuff that we're going to place into the image
ADD target/scala-2.11/docker-spray-http-server-assembly-1.0.jar /app/server.jar

# entry jar to be run in a container
ENTRYPOINT [ "java", "-jar", "/app/server.jar" ]

# HTTP port

Build your project as a single jar file:

DockerSprayHttpServer$ sbt assembly

And now navigate to the project’s folder and run:

DockerSprayHttpServer$ docker build .

This will send the newly created image to the Docker daemon.

Running multiple instances on a single machine

Run the following command to see available docker images :

DockerSprayHttpServer$ docker images

Снимок экрана 2014-12-15 в 23.48.52

a83cda03f529 is not in the repo yet – what we’ve just created. We’re going to run multiple instances from it.

First run the 1-st instance:

DockerSprayHttpServer$ docker run -d -p 10001:8080 a83cda03f529

Note, that we have mapped our 8080–>10001 port.

Now, let’s verify the container is running:

DockerSprayHttpServer$ docker ps

Снимок экрана 2014-12-16 в 0.58.12

We exposed port 8080 for the http-server. When run in Docker-container, it maps our port onto another. On top of this I am  on OS X. Do you remember we set at the beginning 3 env vars? One of them is DOCKER_HOST.

You can actually check this IP as follows:

DockerSprayHttpServer$ boot2docker ip

We need to use this IP address (for OS X, as we are not on Linux).

Let’s test it:

DockerSprayHttpServer$ curl $(boot2docker ip):10001/welcome


You can run as many containers as you want! They are completely isolated. For Scala developers, by they way, I’ve found a nice contribution, you can dockerize your artifacts right from SBT.

Docker: A bird’s-eye view


Docker is a new container technology that primarily allows to run multiple applications on the same old server and operating system. It also makes it easy to package and ship applications. Let’s analyze why Docker can become a replacement of virtual machines and why it is better (or not). Though, undoubtedly it has easy-to-use deployment stuff.


The gist

VM hypervisors emulate virtual hardware and guzzle system resources heavily – each VM has its separated operating system. Conversely, Docker containers use a single shared operating system. Basically, each container has only its application and other dependencies. It is very lightweight as we don’t set up a separate operating system for each container. You can run much more applications (containers) in a single physical instance of hardware unlike VM with Docker. Docker is extremely rapidly getting popular, because It’s light, simple and you can save a great deal of money on hardware and electricity.

There are 2 major parts of operating system (Linux) that ensure the proper work of Docker.


Docker is built on top of LXC and allows to divide up operating system’s Kernel. It means that with Docker it is possible to use only Linux and share a single operating system for all containers. I’ve heard, by the way, that Windows platform has recently adapted Docker too.

The LXC technology (LinuX Containers), which Docker is based from, relies on SELinux and its cgroups to keep everyone in their own sandbox. You can restrict hardware resources: CPU, memory, disk. It can also manage the networking and mounts of filesystem for you as well. In other words you can run multiple isolated Linux containers on one host. Linux Containers serve as a lightweight alternative to VMs as they don’t require the hypervisors. Unlike VMs, LXC don’t require the hypervisor – this is not VM, but an ordinary host.

Assume, you have a container image of 1 GB. If you want to use virtualization, you will need 1 GB multiplied by a number of VMs required. With LXC you just share 1GB. Having 1 thousand containers requires just a little over 1GB of space for the containers OS, assuming they are all running the same OS image. Furthermore, LXCs have have better performance compared to VMs.


Another technology used in Docker is AuFS –  advanced multi layered unification filesystem. Docker can save images for you and they can be represented as incremental updates to the baseline snapshots. You may have a base snapshot of e.g. Linux, make minor changes to it, save a new snapshot as a diff. AuFS allows incrementally merge all these snapshot updates into a single layered snapshot. That is the idea.

When to use Docker

If you need a full isolation of resources, a virtual machine is the way to go. However, if you need to run hundreds of isolated processes on an average host, then Docker is a good fit. There are some use cases when Docker is a win. I can see it as a good tool for development where you run tons of containers on a single machine, doing some interesting stuff in parallel with different deployments from a single Docker image. E.g. you can run a large number of development environments on the same host. For example, Dev, QA, Performance environments on old hardware also. Have a look at Fig project which is intended for these purposes and works along with Docker. It has a straightforward tutorial and super easy commands.


Docker is another technique for performing the same tasks as in Virtual Machines but with the restriction of a shared OS Linux. It  is not quite mature however. There might be potential exploits to crash operating system bringing down all of the processes at once. This is the main drawback of Containers, whereas VMs ensure complete isolation. Besides that many companies use one in Production you might want to wait, but you can definitely start playing around with it on your dev continuous integration process.



Containers and Docker: How secure are they 

PaaS under the hood


Getting started with Docker Orcherstration using FIG

The flip side of rule engines on example of Drools and some valuable tips

Hey there! This post is devoted to business rule engines and solely written based on my research and experience with Open-Source Drools engine that has been used in Production for more than 1 year in mission-critical systems. Undoubtedly, this engine is great for many features. But, I want to explain you what you should think of before using it or when not to use it all. Because it is very important step, once you adapted it in your project, it will be quite costly to remove one later.  I am going to be quick on the theory here, pointing you at issues and important things. If you don’t understand something, feel free to ask me but first please refer to the official documentation. I’ll try to underline the major issue I’ve come across, surely there are many more.

A rule engine is not a magical box that does something new . It is intended to be a tool that provides a higher level of abstraction so you can focus less on reinventing the wheel.

On one of our projects we had a critical system with thousands of small “if then else if…” requirements from business that were initially developed as a sequence of checks on java. Gradually, we realized how this clutter turned into a very difficult mess to maintain. Hence we made a decision to apply some open source rule engine framework. After some high-level research Drools framework became our choice, primarily because it is free and open source. No one had experience on the team with commercial rule engines, however.

Ok, let’s get to it!


The order of rules execution

Rule-engines are a good fit if you don’t know the order in which rules are executed, e.g. the order doesn’t matter. Frequently, in order to understand this, you have to decompose your logic in such a way that you have separated simple flat rules not depending on each other.  In a nutshell, Rule engine is like a black box that have inputs and outputs. This is a necessary but not sufficient condition. Besides, when rules change objects data, that are used in other rules, it greatly increases complexity. Hence you always should be vigilant on the order of execution of different rules.

Otherwise, if the order is determinate, frequently it would be better-off using BPM-engines or something else.

Clear domain model with stable public API

Rule-engines are not a fit if your system doesn’t have a clear domain model. Most non-mature systems don’t have one. It’s evolution, sometimes requiring to understand with time. Even clear requirements not always so clear in practice. Why am I saying this? Well,  because a good rule of thumb it that:

“Rules should be working with only the Domain Model of your system with clear public API around it. If they don’t, your rules ultimately will turn into unmanageable mess. Period!”

It seems to be natural, but many developers don’t realize it. Imagine, what happens if you rules start using internal code or other API that is not a part of domain logic.  With such a disgusting approach, the rules will be aware of more then they need to, like internal implementations/behaviors  and other dependencies = tightly coupled code. It’s just like in OOP or SOA principles but in terms of rules. Tightly coupled code/rules is very difficult to change. Needless to say, if this code is not encapsulated. Eventually, you will have to modify your code triggering broken rules very often, when business requirements change.

Indepotence or exactly once

All rules must run multiple times without errors yielding the same result. It’s like writing a bash-script and also considered a good rule of thumb. Otherwise you may encounter lots of misbehaving weird things.

Forget Debugging tools, %100 coverage with tests is needed

Rule-engines are difficult to refactor, you would have to analyze them and/or keep the whole graph in mind if there are dependencies. In terms of Drools it is impossible to trace rules line-by-line. Drools-rules are declarative and don’t have a mechanism to debug them. Therefore, you should ideally cover 100% of your rules with JUnit-tests. I can give you a hint to use Spock-framework on top of Groovy as a Behavioural-Driven Development stuff, where each rule is simply covered with a test. Tests usually come directly from business specs. Each rule should contain only 1 simple thing as minimal as possible.

Cost of change

No one has any idea if there are conflicting rules when a new one is added or existing one is changed.  A complete regression test must be done when rules are changed. The Memory is sometimes a big issue too. And you can do very little to reduce the consumption – you use 3-rd party framework.

Some rules need to be triggered twice or more

Somebody of you can claim that Drools has a feature of rule-invalidation of Facts, but this again turns into unobvious, tangled mess getting in the way to quickly understand why rules yielded such a result. It’s up to you, but you never know whether you will have to flush/invalidate rules in the long run.

Rete Algorithm vs your Domain API and traversals

The majority of modern rule engines including Drools work upon so called Rete Algorithm. Make sure that your codebase, which your rules call, can be built by a Rete algorithm in the working memory. If it’s not, tree traversal of your codebase within rules on such a model will be a big performance issue!  Bear in mind, if you have internal traversal logic inside your public domain API (e.g. some of your methods do traversal for you) happening on LHS (Left hand side operation) clause , it is very bad for Drools’ performance as latter has its own flexible querying model directly related to Rete-tree. Drools itself should be able to build the tree for you based on your API and efficiently traverse it.

Performance and external systems

What if you are required to save a set of operations to a database and each operation must be checked via rule engine. And the next operation depends on the result of the previous one. This imposes performance problems. All operation could be validated and afterwards saved in 1 go. Think it through in advance. I know there are Rule Groups in Drools, but my point – it’s very complicated.

Rules Centralization

The same business rules may apply across different services, leading to redundancy and governance challenges. It imposes a burden upon us to keep the content of those rules in synch over time.  Thus, it is usually recommended to move them to a new dedicated rule service. The downside is that such a service becomes an additional architectural dependency for other services. The responsibility to build and maintain centralized rule service can introduce various headaches. When we need to change currently active rules or introduce new ones we need to ensure that these changes to the existing rule architecture do not have negative or unforeseen impacts, the worst of which can be a cascading effect that causes exceptions that compound across multiple rules (and multiple services), which should be thoroughly tested.

And please! Make sure that your services know nothing about rules but only talk to the service via a clear contract.


This write-up only scratched the surface of issues coming from my experience. To be honest with you, I would say “No” to applying rule engine frameworks in my future projects to a large number of systems. Initially it seems to be pretty easy, but eventually you realize, that third-party rule engines are more of a pain that benefit, especially when it comes to interdependencies and a domain model. Project maintenance becomes very costly due to this.  It’s not as simple as coding a bunch of if statements. The rules engine gives you the tools to handle rule relationships, but you still have to be able to imagine all of that in your mind.

The most important thing, it is quite difficult to use a rule engine effectively and correctly. A team should have solid knowledge of how to use it and understanding with implementation of business model should be mature. Rule engines work well if there is a really flat independent logic among rules/facts. Each framework restricts you to use their specific model. It’s not worth it!

Project Euler: a list of interesting problems


If you are not aware a website called Project Euler has hundreds of algorithmic problems. Despite that most of them are related to math it’s a good resource to warm up/train your brain in coding. You can use any programming language that you want and track progress.

Here’s a list of interesting Euler’s problems in terms of diversity from my point of view with the aim to improve not only math but also programming skils (No/ Problem Titile):

11 – Largest product in a grid
12 – Highly divisible triangular number
15 – Lattice paths
24 – Lexicographic permutations
54 – Poker hands
59 – XOR decryption
62 – Cubic permutations
67 – Maximum path sum II
68 – Magic 5-gon ring
78 – Coin partitions
79 – Passcode derivation
81 – Path sum: two ways
86 – Cuboid route
94 – Almost equilateral triangles
96 – Sudoku
100 – Arranged probability
107 – Minimal network
109 – Darts
114 – Counting block combinations I
115 – Counting block combinations II
116 – Red, green or blue tiles
117 – Red, green, and blue tiles
145 – How many reversible numbers are there below one-billion?
148 – Exploring Pascal’s triangle
150 – Searching a triangular array for a sub-triangle having minimum-sum
154 – Exploring Pascal’s pyramid
165 – Intersections
166 – Criss Cross
181 – Investigating in how many ways objects of two different colours can be grouped
182 – RSA encryption
186 – Connectedness of a network
194 – Coloured Configurations
208 – Robot Walks
209 – Circular Logic
232 – The Race
267 – Billionaire
275 – Balanced Sculptures
280 – Ant and seeds

Note that it is highly recommended to solve all Euler’s problems one by one because solving a previous problem has a clue to the next ones.

Command And Query Responsibility Segregation and Event Sourcing: What you should think about in advance

After quite a while this article has finally come out! I started to write it about 6 months ago but because of many factors in my life I procrastinated. This text is mainly devoted to those who want to see major pros and cons of this model. You won’t find here the nuts and bolts however. For this, move to the reference section where listed really cool URLs for the start. Further the pattern will be named as CQRS/ES for simplicity.

2 main goals of the pattern

Some developers consider that this pattern is solely used for scalability reasons forgetting that one greatly reduces data model complexity.

Events and snapshots – no removal but always append

it is well known that append-only architectures distribute more easily than updating architectures because there are far fewer locks to deal with. Append-only operations require no locks on a database-side which increases both performance and scalability.

CQRS + Event Model

One of the largest issues when using Event Sourcing is that you cannot ask the system a query such as “Give me all users whose first names are ‘Greg’”. This is due to not having a representation of current state.

With Event Sourcing the event model is also the persistence model on the Write side. This drastically lowers costs of development as no conversion between the models is needed.

Synchronizing write and read sides

The choice of integration model though is very important as translation and synchronization between models can be become a very expensive undertaking. The model that is best suited is the introduction of events, events are a well known integration pattern and offer the best mechanism for model synchronization.

CQRS suggests using 2 different databases: one optimized for reading (denormalized) and another for writing. It is a very tempting way to designing systems. The only big problem with this solution is that you need somehow to sync up your 2 data storages.

You must definitely have a mechanism to recreate a read-side from the write-side. You might need to do this if the read side data store got out of synchronization for some reason, or because you needed to modify the structure of the read-side data store to support a new query. Remember that the CQRS pattern does not mandate that you use different stores on the read side and write side. You could decide to use a single relational store with a schema in third normal form and a set of denormalized views over that schema. However, replaying events is a very convenient mechanism for resynchronizing the read-side data store with the write-side data store.

Storage technology replaceability

CQRS/ES makes it easy to change your technologies. For example, you could start with a file-based event store for prototyping and development, and later switch to a Windows Azure table-based store for production.

Relational vs NoSQL solutions and the cost of maintenance – think twice

Let’s mention major issues in case of using a relational database as a technology for this model.

Pitfall 1: Limit in number of columns per table

Read-side is always kept in a denormalized form for performance reasons and usually requires a huge number of columns. If you have e.g. Oracle 11g database and hit a maximum number of 1000 columns per table you won’t be able to add more columns.

Pitfall 2: Adding new columns

In many cases in order to add new data to the domain model you must release a bunch of applications/services along with changing a schema of a database on the read-side, especially when a column is not empty but already has some data. Here, I prefer a NoSQL databases such as MongoDB with relaxed schema that doesn’t require to change one (data are kept in JSON documents) thus you won’t have to change the schema of a database at all! CQRS/ES patterns with NoSQL db might have a simple Event Storage for the write-side with a blob-column in which data are represented as history and read-side as a schema-less NoSQL db.

Eventual consistency – think twice

While not required, it is common to use messaging between the write and read sides. This means that the system will be in an inconsistent state from time to time. This pattern is not a fit for e.g. financial transactions where a strong consistency is required between 2 sides. However, in majority of cases strong consistency can be used only on the write-side whereas the read-side is used for querying and reporting.

Simple vs costly

Events are simple objects that describe what has happened in the system. By simply saving events, you are avoiding the complications associated with saving complex domain objects to a relational store; namely, the object-relational impedance mismatch.

The event based model may be slightly more costly due to the need of definition of events but this cost is relatively low and it also offers a smaller Impedance Mismatch to bridge which helps to make up for the cost. The event based model also offers all of the benefits discussed in “Events” that also help to reduce the overall initial cost of the creation of the events. That said the CQRS and Event Sourcing model is actually less expensive in most cases. That said the CQRS and Event Sourcing model is actually less expensive in most cases.


As long as you have a stream of events, you can project it to any form, even a conventional SQL database. For instance, my favorite approach is to project event streams into JSON documents stored in a cloud storage.

Also a great benefit of the separate read layer is that it will not suffer from an impedance mismatch – It is connected directly to the data model, this can make queries much easier to optimize. On the write side you no longer need to worry about how your locks impact queries, and on the read side your database can be read-only.



Because events are immutable, you can use an append-only operation when you save them. Events are also simple, standalone objects. Both these factors can lead to better performance and scalability for the system than approaches that use complex relational storage models. On top of this events reduce network volumes unlike full snapshots in case of a pure CQRS model.


A normalized database schema can fail to provide adequate response times because of the excessive table JOIN operations. Despite advances in relational database technology, a JOIN operation is still very expensive compared to a single-table read.


Command: In most systems, especially web systems, the Command side generally processes a very small number of transactions as a percentage of the whole. Scalability therefore is not always important.

Query: In most systems, especially web systems, the Query side generally processes a very large number of transactions as a percentage of the whole (often times 2 or more orders of magnitude). Scalability is most often needed for the query side.

Distributed development teams

The architecture can be viewed as three distinct decoupled areas. The first is the client; it consumes DTOs and produces Commands. The second is the domain; it consumes commands and produces events. The third is the Read Model; it consumes events and produces DTOs. The decoupled nature of these three areas can be extremely valuable in terms of team characteristics. Instead of working in vertical slices the team can be working on three concurrent vertical slices, the client, the domain, and the read model. This allows for a much better scaling of the number of developers working on a project as since they are isolated from each other they cause less conflict when making changes. It would be reasonable to nearly triple a team size without introducing a larger amount of conflict due to not needing to introduce more communication for the developers.


There is no one right way to implement CQRS. It is not recommended to use this model for critical projects of course if you have no experience with one. Note that It is not possible to create an optimal solution for searching, reporting, and processing transactions utilizing a single model. 


Introduction to Domain Driven Design, CQRS and Event Sourcing

Command-Query-Responsibility-Segregation (CQRS) Makes Domain Models Practical

CQRS and Event Sourcing

CQRS – Did you mean CARS?

A CQRS and ES Deep Dive

Clarified CQRS