Laravel Multinenacy: Jobs

UPDATE 2019-02-01: Mohamed Said wrote a post on Diving Laravel about managing multi-tenancy database connections in Laravel. He also added a section in which it's described how to do the same as I explain here but with two lines of code inside your AppServiceProvider 🤯. So please, read this post if you want to learn something new. But if your case is pretty the same as mine, follow Said's example as I consider this how-to deprecated. We've switched to Mohamed's solution at work with excellent results 👍.

UPDATE 2019-01-28: Since database driver is made just for local development purposes, my team and I switched to Redis in production and made similar modifications to the Redis classes. I'm planning to write more about this topic in a new post.


Our team is working hard on a project which can connect to different databases depending on the client. Here is the wikipedia link: multi-tenancy:

The term "software multitenancy" refers to a software architecture in which a single instance of software runs on a server and serves multiple tenants.

So the point here is that we have a single installation of our app and depending on the user logged in, it will show the respective data.

When you send a job to a queue, Laravel serializes the class to save it using your chosen driver until a Worker can fire it. When the job is fired, it tries to restore the models you passed in the constructor from the database using the SerializesModel trait.

This works totally fine, but sometimes you get a non-trivial requirement, and this time it was our turn.

We don't have a connection defined for each client in config/databases.php configuration file. Instead, we have the database name in a clients table with the other information, and each User has a client_id to create the relation. We use a middleware at the beginning of each request to set the connection attributes in the configuration.

The problem began when we started working with jobs: due to the fact that there's no client logged in, there's no way to determine which connection must be set up. After an intense research, and digging through the framework's code, we figured out a solution which fitted our needs.

I'm going to show some examples with the database driver, since it's the one we chose for our app. But I'm sure you can apply the same modifications to your driver of choice.

Our goal here is to set up each job with the connection it needs, so the job must know which client it belongs to.

This step is easy: since we use the database driver, we just need to run this command:

php artisan queue:table

Then we modify the migration file and we add our 'client_id' before migrating

Schema::create('jobs', function (Blueprint $table) {
    $table->bigIncrements('id');
    $table->string('queue')->index();
    $table->integer('client_id')->unsigned()->nullable();
    ...
});

But how will the framework know the way to correctly populate this new field whenever it dispatches a new job? This is where the tricky part starts:

In our config/app.php, we need to specify our custom QueueServiceProvider instead of Illuminate's:

'providers' => [
    ...
    Illuminate\Pipeline\PipelineServiceProvider::class,
    App\Queue\QueueServiceProvider::class,
    Illuminate\Redis\RedisServiceProvider::class,
    ...

From now on, we will organize the extended classes of Illuminate's Queue in the App\Queues namespace

This provider is very simple, it just needs to extends the original class.

namespace App\Queue;

use App\Queue\Connectors\DatabaseConnector;

class QueueServiceProvider extends \Illuminate\Queue\QueueServiceProvider
{
    protected function registerDatabaseConnector($manager)
    {
        $manager->addConnector('database', function () {
            return new DatabaseConnector($this->app['db']);
        });
    }
}

We need to override DatabaseConnector in order to use our own, which overrides connect to use our DatabaseQueue

namespace App\Queue\Connectors;

use App\Queue\DatabaseQueue;

class DatabaseConnector extends \Illuminate\Queue\Connectors\DatabaseConnector
{

    public function connect(array $config)
    {
        return new DatabaseQueue(
            $this->connections->connection($config['connection'] ?? null),
            $config['table'],
            $config['queue'],
            $config['retry_after'] ?? 60
        );
    }
}

Now we need to override the method buildDatabaseRecord to check if we have a client and, if we do, fill up that field on the jobs table.

namespace App\Queue;

use App\Queue\Jobs\DatabaseJob;

class DatabaseQueue extends \Illuminate\Queue\DatabaseQueue
{

    protected function buildDatabaseRecord($queue, $payload, $availableAt, $attempts = 0)
    {
        $queueRecord = [
            'queue' => $queue,
            'attempts' => $attempts,
            'reserved_at' => null,
            'available_at' => $availableAt,
            'created_at' => $this->currentTime(),
            'payload' => $payload,
        ];

        if (client()) {
            $queueRecord['client_id'] = client()->id;
        }

        return $queueRecord;
    }
}

Okay, so now our jobs have a client_id assigned, the only remaining thing is... How do we set up the client connection before a job is fired? Yes, you guessed it, we will need to extend another Illuminate class for that

namespace App\Queue\Jobs;

use App\Models\Client;
use App\Helpers\ClientConnector;

class DatabaseJob extends \Illuminate\Queue\Jobs\DatabaseJob
{
    public function fire()
    {
        if ($this->job->client_id) {
            $client = Client::findOrFail($this->job->client_id);

            ClientConnector::connectByClient($client);
        }

        parent::fire();
    }
}

Lastly, we need to come back to our own DatabaseQueue and tell it to use our DatabaseJob instead of Illuminate's Queue default by overriding the method marshalJob

protected function marshalJob($queue, $job)
{
    $job = $this->markJobAsReserved($job);

    return new DatabaseJob(
        $this->container, $this, $job, $this->connectionName, $queue
    );
}

You may also like: Laravel Multitenancy: Route Model Binding

Found a typo? Edit this post.
Special thanks to Dieter for the amazing code of this blog.

Got any hints?
No, I'm afraid not.
However if you search within yourself, then...
Cut it out, ok?
I asked for a hint not the crazy philosophies of a Father Christmas look-a-like.
Heed my words, bold adventurer, and your path will lead to wisdom and..
There's just no reasoning with him...
I'm totally stuck as what to do.
I really wish I could help you but I'm just a clueless old fool.
Never a truer word was spoken.
Still clueless?
I'm afraid so.
Still look like a Father Christmas?
I'm afraid so.
Good.
Calypso from Fleur de Lys Simon The Sorcerer