Shopware 5 – High-Speed Parallel Thumbnail Generation

Shopware 5 - High-Speed Parallel Thumbnail Generation

How to tune the integrated SW5 mechanism to work more then 22 times faster!

2021-04-29 09:24:11

Shopware 5 – Parallel Thumbnail Generation After Moving a Shopware 5 System to Another Server

We hat a customer with 400k images and 1600k thumbnails that needed to move from an old hdd based server to a new ssd based one. The problem was that the old server was so slow that it already needed 2 days to count through all images, not speaking about coping them.

So we decided to copy only the original images and regernerate the Tumbnails. For copying the original images, i created a small console command that exports all paths of the original images that we need to copy: ExportImagesCommand.php

This filelist can be used with tar -T or rsync –files-from= options that tells these tools to only process the listed files. for the initial copy process, tar is highly recommended, as it just picks up the files listed without doing any „calculation“ as rsync does.

Shopware 5 – Parallel Thumbnail Generation After Moving a Shopware 5 System to Another Server

We had a customer with 400k images and 1600k thumbnails that need to be moved from an old hdd based server to a new ssd based one. The problem was that the old server was so slow that it already needed 2 days counting through all images, not speaking about copying them.

So we decided to copy only the original images and regenerate the Thumbnails. For copying the original images, I created a small console command that exports all paths of the original images that we need to copy: ExportImagesCommand.php

This file list can be used with tar -T or rsync –files-from= options that tell these tools to only process the listed files. For the initial copy process, tar is highly recommended as it just picks up the files listed without doing any „calculation“ as rsync does.

SW5 Default Thumbnail Generation Would Have Taken 80 Hours

… and would only use half of a core.

I was curious if I can speed up this generation process. The server itself has 32 cores available, so I copied the generate thumbnail command from sw5 and modified it to work in batches with an –batch parameter:

ParallelThumbnailGenerateCommand.php

To make it work, I just modified the Shopware core at engine/Shopware/Models/Media/Repository.php

I just changed the getAlbumMediaQuery function to:

public function getAlbumMediaQuery($albumId, $filter = null, $orderBy = null, $offset = null, $limit = null, $validTypes = null, $batch = null)
{
    $builder = $this->getAlbumMediaQueryBuilder($albumId, $filter, $orderBy, $validTypes);
    if (is_numeric($batch)) {
        $builder->andWhere('MOD(media.id, 1000) = ?3');
        $builder->setParameter(3, $batch);
    }

    if ($limit !== null) {
        $builder->setFirstResult($offset)
                ->setMaxResults($limit);
    }

    return $builder->getQuery();
}

It is an optional parameter and won’t break anything. If you do a Shopware update, this would be gone, but as I was in the quest to speed up things for a one-time task, I just modified it in the core instead of finding a long-term solution.

What this function does is calculating a modulo of 1000 on the media id and compares it with the batch id. So we basically have 1000 batches to process until all work is done.

Now we only need to start all 1000 batches in a parallel manner. For doing so, I used the very helpful tool parallel – which is available in Linux:

parallel -j 64 ./bin/console my:image:generate:thumbnails --batch ::: {0..999}

It starts 64 batches in parallel and continues its work until all 1000 batches are finished.

And this is how it looks at htop:

:Linux htop console view with 32 CPU core  working diagramm (all cores are used) and text conole output of SW5 Thumbnail Generation Batches currently running

Finally, all the work is now finished in 3.5 hours instead of 80.

Some Questions?

Don’t hesitate to contact us. We will answer all your questions as soon as possible.