In the previous article we explored why sharding by tenant is a very good solution for us. In this article we dig into how to divide our Symfony app by shard.
Doctrine
We chose Doctrine as our ORM so let’s see what it offers to us.
First of all you have to note that Doctrine is composed mainly by two different parts: DBAL and ORM. ORM leverages DBAL functionalities and it’s completely transparent for sharding. DBAL is the layer we’re interested in. Please note we’re using DBAL 2.5.12 as the moment of writing.
Doctrine DBAL
It offers natively the sharding functionality but you need to configure it and, if required, make some customizations. Your starting point should be the Doctrine sharding documentation. It’s the doctrine doc and not the Symfony one. But we use Symfony, so we need to dig also into the Doctrine Bundle to find out how DBAL should be configured. For the full options’ list have a look at the Doctrine bundle configuration reference.
Sharding configuration
In our project sharding is set up on the default connection. This means the default connection will be tied to a chosen shard at a given point in time. As you probably remember from previous articles, we want to always have a fixed connection to the global database. So, we configure 2 connections: default and global.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
doctrine: dbal: connections: default: shard_choser_service: nuvola.app.sharding.shard_choser shard_manager_class: Nuvola\AppBundle\Sharding\ShardManager charset: UTF8 dbname: '%database_name%' driver: '%database_driver%' host: '%database_host%' password: '%database_password%' port: '%database_port%' user: '%database_user%' shards: shard1: id: 1 host: '%database_host%' user: '%database_user%' password: '%database_password%' dbname: dbshard1 charset: UTF8 shard2: id: 2 host: '%database_host%' user: '%database_user%' password: '%database_password%' dbname: dbshard2 charset: UTF8 global: charset: UTF8 dbname: '%database_name%' driver: '%database_driver%' host: '%database_host%' password: '%database_password%' port: '%database_port%' user: '%database_user%' default_connection: default |
Please be aware that the global connection has nothing to do with sharding. It’s a simple connection pointing to the global db. The default connection, on the contrary, has three new keys: “shards”, “shard_choser_service” and “shard_manager_class”.
Shards
“Shards” key represents an array of shards. This key will represent the so called “distribution value”, which is the name Doctrine uses to reference a shard. Each shard must have a unique id and the connection parameters. Because the default connection has the “shards” key, Doctrine will use “Doctrine\\DBAL\\Sharding\\PoolingShardConnection” to manage the default connection. If you want you can override it and make your customizations: we haven’t needed to do it.
Shard chooser
How can Doctrine retrieve shard id given a distribution value? The answer is the shard chooser service.
You have to create a class implementing “\Doctrine\DBAL\Sharding\ShardChoser\ShardChoser”.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 |
<?php namespace Nuvola\AppBundle\Sharding; use Doctrine\DBAL\Sharding\PoolingShardConnection; use Symfony\Component\Filesystem\Filesystem; use Symfony\Component\Yaml\Yaml; class ShardChoser implements \Doctrine\DBAL\Sharding\ShardChoser\ShardChoser { /** * @var string */ private $shardsFile; /** * @var string */ private $cacheFile; /** * @param string $shardsFile * @param string $cacheDir */ public function __construct($shardsFile, $cacheDir) { $this->shardsFile = $shardsFile; $this->cacheFile = $cacheDir . DIRECTORY_SEPARATOR . 'sharding' . DIRECTORY_SEPARATOR . basename($this->shardsFile, '.yml') . '.cache.php'; } /** * {@inheritdoc} */ public function pickShard($distributionValue, PoolingShardConnection $conn) { $distributionValue = strtolower($distributionValue); $shards = $this->getShards($conn); if (array_key_exists($distributionValue, $shards)) { return $shards[$distributionValue]['id']; } throw new \DomainException( sprintf( 'Unable to find shard for "%s"', $distributionValue )); } /** * @throws \InvalidArgumentException * * @return array */ public function getShards(PoolingShardConnection $conn) { if (file_exists($this->cacheFile)) { $info = include $this->cacheFile; return $info; } if (!is_file($this->shardsFile)) { throw new \InvalidArgumentException('Unable to load file ' . $this->shardsFile); } $shardsConfiguration = Yaml::parse(file_get_contents($this->shardsFile)); $info = $shardsConfiguration['doctrine']['dbal']['connections']['default']['shards']; $cacheContent = '<?php return ' . var_export($info, true) . ';'; $fs = new Filesystem(); $fs->dumpFile($this->cacheFile, $cacheContent); return $info; } } |
The “pickShard” method reads our Doctrine config file and gets back the shard id given the distribution value. We want to avoid to parse the config file every time and so we parse it the first time using the Yaml component and then we simply read a php array. You should have looked at the public qualifier for the “getShards” method. We call it from other services (more on following articles) so we cannot make it private.
Another interesting bit to spot is the “$shardsFile” constructor argument. If you have multiple environments (dev, prod, test) your shards are different and you have a config file for each environment. Passing it to the constructor you can be sure to read from the proper file.
Here is the service definition:
|
1 2 3 4 5 6 |
<service id="nuvola.app.sharding.shard_choser" class="Nuvola\AppBundle\Sharding\ShardChoser" public="false"> <argument>%nuvola.shards_config_file%</argument> <argument>%kernel.cache_dir%</argument> </service> |
Shard manager
By default Doctrine uses the “Doctrine\\DBAL\\Sharding\\PoolingShardManager” class as shard manager for the “default” connection. We want to use a custom one and Doctrine allows you to do that by simply creating a class which implements the “\Doctrine\DBAL\Sharding\ShardManager” interface.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 |
<?php namespace Nuvola\AppBundle\Sharding; use Doctrine\DBAL\Sharding\PoolingShardConnection; class ShardManager implements \Doctrine\DBAL\Sharding\ShardManager { /** * @var PoolingShardConnection */ private $poolingShardConnection; /** * @var ShardChoser */ private $shardChoser; /** * @var string|null */ private $currentDistributionValue; /** * @param PoolingShardConnection $conn */ public function __construct(PoolingShardConnection $conn) { $params = $conn->getParams(); $this->poolingShardConnection = $conn; $this->shardChoser = $params['shardChoser']; } /** * {@inheritdoc} */ public function selectGlobal() { $this->poolingShardConnection->connect(0); $this->currentDistributionValue = null; } /** * {@inheritdoc} */ public function getCurrentDistributionValue() { return $this->currentDistributionValue; } /** * {@inheritdoc} */ public function selectShard($distributionValue = null) { try { if (null == $distributionValue) { $this->selectGlobal(); } else { $shardId = $this->shardChoser->pickShard($distributionValue, $this->poolingShardConnection); $this->poolingShardConnection->connect($shardId); $this->currentDistributionValue = $distributionValue; } } catch (\Exception $e) { throw new ShardNonSelezionabileException(sprintf( 'Error selecting shard "%s"', $distributionValue )); } } /** * {@inheritdoc} */ public function getShards() { return $this->shardChoser->getShards($this->poolingShardConnection); } /** * {@inheritdoc} */ public function queryAll($sql, array $params, array $types) { $shards = $this->getShards(); if (!$shards) { throw new \RuntimeException('No shards found.'); } $result = []; $oldDistribution = $this->getCurrentDistributionValue(); foreach ($shards as $shard) { $this->poolingShardConnection->connect($shard['id']); foreach ($this->poolingShardConnection->fetchAll($sql, $params, $types) as $row) { $result[] = $row; } } if ($oldDistribution === null) { $this->selectGlobal(); } else { $this->selectShard($oldDistribution); } return $result; } } |
Sharding and slaves
If you want to have slave connections for your shard Doctrine doesn’t offer this functionality at the time of writing. To dig more into see https://github.com/doctrine/dbal/issues/2737
In a following article we’ll see how our Symfony controllers and commands can leverage this sharding infrastructure. If you like this kind of things… why not having a look of our team? We’re always looking for smart and talented people!
[…] Doctrine Sharding […]