Via G. Falcone 5, Pollenza (MC), Italy
+39 0733 203595

Doctrine sharding

In the previous article we explored why sharding by tenant is a very good solution for us. In this article we dig into how to divide our Symfony app by shard.

Doctrine

We chose Doctrine as our ORM so let’s see what it offers to us.

First of all you have to note that Doctrine is composed mainly by two different parts: DBAL and ORM. ORM leverages DBAL functionalities and it’s completely transparent for sharding. DBAL is the layer we’re interested in. Please note we’re using DBAL 2.5.12 as the moment of writing.

Doctrine DBAL

It offers natively the sharding functionality but you need to configure it and, if required, make some customizations. Your starting point should be the Doctrine sharding documentation. It’s the doctrine doc and not the Symfony one. But we use Symfony, so we need to dig also into the Doctrine Bundle  to find out how DBAL should be configured. For the full options’ list have a look at the Doctrine bundle configuration reference.

Sharding configuration

In our project sharding is set up on the default connection. This means the default connection will be tied to a chosen shard at a given point in time. As you probably remember from previous articles, we want to always have a fixed connection to the global database. So, we configure 2 connections: default and global.

Please be aware that the global connection has nothing to do with sharding. It’s a simple connection pointing to the global db. The default connection, on the contrary, has three new keys: “shards”, “shard_choser_service” and “shard_manager_class”.

Shards

“Shards” key represents an array of shards. This key will represent the so called “distribution value”, which is the name Doctrine uses to reference a shard. Each shard must have a unique id and the connection parameters. Because the default connection has the “shards” key, Doctrine will use “Doctrine\\DBAL\\Sharding\\PoolingShardConnection” to manage the default connection. If you want you can override it and make your customizations: we haven’t needed to do it.

Shard chooser

How can Doctrine retrieve shard id given a distribution value? The answer is the shard chooser service.

You have to create a class implementing “\Doctrine\DBAL\Sharding\ShardChoser\ShardChoser”.

 

The “pickShard” method reads our Doctrine config file and gets back the shard id given the distribution value. We want to avoid to parse the config file every time and so we parse it the first time using the Yaml component and then we simply read a php array. You should have looked at the public qualifier for the “getShards” method. We call it from other services (more on following articles) so we cannot make it private.

Another interesting bit to spot is the “$shardsFile” constructor argument. If you have multiple environments (dev, prod, test) your shards are different and you have a config file for each environment. Passing it to the constructor you can be sure to read from the proper file.

Here is the service definition:

Shard manager

By default Doctrine uses the “Doctrine\\DBAL\\Sharding\\PoolingShardManager” class as shard manager for the “default” connection. We want to use a custom one and Doctrine allows you to do that by simply creating a class which implements the “\Doctrine\DBAL\Sharding\ShardManager” interface. 

Sharding and slaves

If you want to have slave connections for your shard Doctrine doesn’t offer this functionality at the time of writing. To dig more into see https://github.com/doctrine/dbal/issues/2737

In a following article we’ll see how our Symfony controllers and commands can leverage this sharding infrastructure. If you like this kind of things… why not having a look of our team? We’re always looking for smart and talented people!

 

1 comment

Leave a reply