The problem
When a software handles multiple concurrent operations an entire class of problems arises.
In our case multiple requests would try to update centralized information leading to unique constraint violations at the database level.
That was because the code would check if an entity to update already existed and created a new instance if it wasn’t found.
This time between checking and creation of the new entity causes a race condition because another request may perform the check during this time causing two creation attempts.
To solve this problem the check-and-create operation needs to be atomic. In other words no other operation on this resource can be performed while it is in progress.
The solution
A simple and effective way to do this is the use of locks.
Symfony already provides a ready to use lock component.
To be able to avoid simultaneous executions of the critical section (in this case the code that checks that an entity exists and creates a new one if it doesn’t) the component needs a centralized store to manage held locks.
Having a multiple webserver production environment we needed a remote store because a local store would not prevent two requests on different servers to run the critical section at the same time.
So our choice was the Redis store mainly because we already use it in our setup.
The code
The only remaining problem was the fact that this store doesn’t support blocking, but a convenient decorator is already present in Symfony.
This decorator is RetryTillSaveStore that as the name suggests keeps trying at regular configurable intervals to acquire a lock held by someone else until it succeeds.
All we needed to do was to create a wrapper for the factory to configure the class the way we needed.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
class RedisLockFactory { /** * @var Factory */ private $baseFactory; public function __construct(\Redis $redisClient) { $redisStore = new RedisStore($redisClient); $blockingRedisStore = new RetryTillSaveStore($redisStore, 200); $this->baseFactory = new Factory($blockingRedisStore); } ... |
Our factory has a simple method to create a new lock:
|
1 2 3 4 |
public function createLock(string $resource, float $ttl = 30.0, bool $autoRelease = true): LockInterface { return $this->baseFactory->createLock($resource, $ttl, $autoRelease); } |
The purpose of the resource string is to serialize only conflicting operations so the typical content is a concatenation, with some kind of separator, of entity names, ids or anything that would result the same for conflicting operations.
In our case it was the name of the entity class and the values of the unique constraint fields.
The second parameter specifies the time to live after which locks are released automatically.
That’s in case a process is interrupted while holding a lock because otherwise it would remain locked forever. You can configure it with an order of magnitude over the maximum expected execution time of the critical section as it’s only a safe net when failures occur.
The auto-release flag is a convenient feature that performs the lock release when the lock object is destroyed so if the developer forgets to release the lock this is automatically performed when the variable that references the lock object goes out of scope.
The usual pattern for using this locks on this use case is
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
$lock = $lockFactory->createLock(sprintf( '%s_%s', $uniqueConstraintFieldValue1, $uniqueConstraintFieldValue2 )); $lock->acquire(true); $entity = $repository->findOneByUniqueConstraint($uniqueConstraintFieldValue1, $uniqueConstraintFieldValue2); if (!$entity instanceof MyEntity) { $entity = new MyEntity(/* constructor arguments */); $em->persist($entity); $em->flush(); } $lock->release(); // ... perform entity update ... |
As you can see the check and create operations are enclosed between the lock acquisition and release, so when two requests try to execute this code at the same time one will be granted the lock and perform the operation while the other is blocked inside the acquire method of the lock.
When the first process completes the operation and performs the release, only then the blocked process will be free to acquire the lock avoiding the concurrent attempt to create the entity because the second one tries to find the entity only after the first one has already flushed it.
