Skip to content

Commit d6460cb

Browse files
st3inyst3iny
authored
Merge pull request #599 from nextcloud/feat/background-job
feat: implement background job to generate previews
2 parents 0d24519 + 1659b51 commit d6460cb

18 files changed

+710
-199
lines changed

‎README.md‎

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,13 +42,21 @@ requested before it is pre-generated it will still be shown.
4242

4343
## How does the app work
4444

45-
1. Listen to events that a file has been written or modified and store it in the database
46-
2. On cron run request previews for the files that have been written or modified
45+
1. Listen to events that a file has been written or modified and store it in the database.
46+
2. Generates previews for the files that have been written or modified in a background job.
47+
3. Optional: Dedicated occ command to generate previews using a custom schedule (for example, in a
48+
separate system cron job).
4749

48-
If a preview already exists at step 2 then requesting it is really cheap. If not
50+
If a preview already exists at step 2 (or 3) then requesting it is really cheap. If not
4951
it will be generated. Depending on the sizes of the files and the hardware you
5052
are running on the time this takes can vary.
5153

54+
By default, the background job to generate previews for modified files is limited to a maximum
55+
execution time of five minutes. Additionally, it requires using the cron background job mode.
56+
Webcron and AJAX modes are not supported. The background job is limited to prevent stalling the PHP
57+
process. The limits are configurable via app configs (see below) or admins can configure a dedicated
58+
system cron job which runs the `occ preview:pre-generate` command.
59+
5260
## Commands
5361

5462
#### `preview:generate-all [--workers=WORKERS] [--path=PATH ...] [user_id ...]`
@@ -101,6 +109,19 @@ the aspect ratio.
101109
Will retain the aspect ratio and use the specified height. The width will be scaled according to
102110
the aspect ratio.
103111

112+
#### `occ config:app:set --value=false --type=bool previewgenerator job_disabled`
113+
Set to true to disable the background job that generates previews by default without having to
114+
configure a manual system cron job. It is recommended to disable the default background job in case
115+
a custom system cron entry with `occ preview:pre-generate` is configured (set this config to true).
116+
117+
#### `occ config:app:set --value=600 --type=int previewgenerator job_max_execution_time`
118+
Limits the maximum execution time in seconds of the preview background job. (A value of zero means
119+
unlimited.)
120+
121+
#### `occ config:app:set --value=0 --type=int previewgenerator job_max_previews`
122+
Limits the count of previews to be generated in each execution of the preview background job. (A
123+
value of zero means unlimited.) Configure one, both or no limit (not recommended!). In case both
124+
limits are configured, the more restrictive one takes precedence.
104125

105126
## FAQ
106127

‎appinfo/info.xml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ The first time you install this app, before using a cron job, you properly want
1515
1616
**Important**: To enable pre-generation of previews you must add **php /var/www/nextcloud/occ preview:pre-generate** to a system cron job that runs at times of your choosing.]]>
1717
</description>
18-
<version>5.12.0-dev.2</version>
18+
<version>5.12.0-dev.4</version>
1919
<licence>agpl</licence>
2020
<author>Richard Steinmetz</author>
2121
<namespace>PreviewGenerator</namespace>
@@ -30,9 +30,12 @@ The first time you install this app, before using a cron job, you properly want
3030
<php min-version="8.1" max-version="8.5" />
3131
<nextcloud min-version="30" max-version="34" />
3232
</dependencies>
33-
33+
<background-jobs>
34+
<job>OCA\PreviewGenerator\BackgroundJob\PreviewJob</job>
35+
</background-jobs>
3436
<commands>
3537
<command>OCA\PreviewGenerator\Command\Generate</command>
3638
<command>OCA\PreviewGenerator\Command\PreGenerate</command>
39+
<command>OCA\PreviewGenerator\Command\QueueStats</command>
3740
</commands>
3841
</info>

‎lib/BackgroundJob/PreviewJob.php‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
<?php
2+
3+
declare(strict_types=1);
4+
5+
/**
6+
* SPDX-FileCopyrightText: 2025 Nextcloud GmbH and Nextcloud contributors
7+
* SPDX-License-Identifier: AGPL-3.0-or-later
8+
*/
9+
10+
namespace OCA\PreviewGenerator\BackgroundJob;
11+
12+
use OCA\PreviewGenerator\Service\ConfigService;
13+
use OCA\PreviewGenerator\Service\PreGenerateService;
14+
use OCA\PreviewGenerator\Support\OutputInterfaceLoggerAdapter;
15+
use OCA\PreviewGenerator\Support\PreviewLimiter\CountLimiter;
16+
use OCA\PreviewGenerator\Support\PreviewLimiter\ExecutionTimeLimiter;
17+
use OCA\PreviewGenerator\Support\PreviewLimiter\MultiLimiter;
18+
use OCA\PreviewGenerator\Support\PreviewLimiter\PreviewLimiter;
19+
use OCP\AppFramework\Utility\ITimeFactory;
20+
use OCP\BackgroundJob\TimedJob;
21+
22+
class PreviewJob extends TimedJob {
23+
private readonly PreviewLimiter $limiter;
24+
25+
public function __construct(
26+
ITimeFactory $time,
27+
private readonly PreGenerateService $preGenerateService,
28+
private readonly OutputInterfaceLoggerAdapter $outputInterface,
29+
private readonly ConfigService $configService,
30+
) {
31+
parent::__construct($time);
32+
$this->setInterval(5 * 60);
33+
$this->setTimeSensitivity(self::TIME_SENSITIVE);
34+
35+
$limiters = [];
36+
37+
$maxPreviews = $this->configService->getMaxBackgroundJobPreviews();
38+
if ($maxPreviews > 0) {
39+
$limiters[] = new CountLimiter($maxPreviews);
40+
}
41+
42+
$maxExecutionTime = $this->configService->getMaxBackgroundJobExecutionTime();
43+
if ($maxExecutionTime > 0) {
44+
$limiters[] = new ExecutionTimeLimiter($time, $maxExecutionTime);
45+
}
46+
47+
$this->limiter = new MultiLimiter($limiters);
48+
}
49+
50+
protected function run($argument) {
51+
if ($this->configService->isBackgroundJobDisabled()
52+
|| !$this->configService->usesCronDaemon()) {
53+
return;
54+
}
55+
56+
$this->preGenerateService->setLimiter($this->limiter);
57+
$this->preGenerateService->preGenerate($this->outputInterface);
58+
}
59+
}

‎lib/Command/Generate.php‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ protected function configure(): void {
9999

100100
protected function execute(InputInterface $input, OutputInterface $output): int {
101101
if ($this->encryptionManager->isEnabled()) {
102-
$output->writeln('Encryption is enabled. Aborted.');
102+
$output->writeln('<error>Encryption is enabled. Aborted.</error>');
103103
return 1;
104104
}
105105

‎lib/Command/PreGenerate.php‎

Lines changed: 8 additions & 182 deletions
Original file line numberDiff line numberDiff line change
@@ -9,75 +9,17 @@
99

1010
namespace OCA\PreviewGenerator\Command;
1111

12-
use OC\DB\Exceptions\DbalException;
13-
use OCA\PreviewGenerator\Service\NoMediaService;
14-
use OCA\PreviewGenerator\SizeHelper;
15-
use OCP\AppFramework\Db\TTransactional;
16-
use OCP\AppFramework\Utility\ITimeFactory;
17-
use OCP\DB\Exception;
18-
use OCP\Encryption\IManager;
19-
use OCP\Files\File;
20-
use OCP\Files\GenericFileException;
21-
use OCP\Files\IRootFolder;
22-
use OCP\Files\NotFoundException;
23-
use OCP\IConfig;
24-
use OCP\IDBConnection;
25-
use OCP\IPreview;
26-
use OCP\IUserManager;
12+
use OCA\PreviewGenerator\Exceptions\EncryptionEnabledException;
13+
use OCA\PreviewGenerator\Service\PreGenerateService;
2714
use Symfony\Component\Console\Command\Command;
2815
use Symfony\Component\Console\Input\InputInterface;
2916
use Symfony\Component\Console\Output\OutputInterface;
3017

3118
class PreGenerate extends Command {
32-
use TTransactional;
33-
34-
/* @return array{width: int, height: int, crop: bool} */
35-
protected array $specifications;
36-
37-
protected string $appName;
38-
protected IUserManager $userManager;
39-
protected IRootFolder $rootFolder;
40-
protected IPreview $previewGenerator;
41-
protected IConfig $config;
42-
protected IDBConnection $connection;
43-
protected OutputInterface $output;
44-
protected IManager $encryptionManager;
45-
protected ITimeFactory $time;
46-
protected NoMediaService $noMediaService;
47-
protected SizeHelper $sizeHelper;
48-
49-
/**
50-
* @param string $appName
51-
* @param IRootFolder $rootFolder
52-
* @param IUserManager $userManager
53-
* @param IPreview $previewGenerator
54-
* @param IConfig $config
55-
* @param IDBConnection $connection
56-
* @param IManager $encryptionManager
57-
* @param ITimeFactory $time
58-
*/
59-
public function __construct(string $appName,
60-
IRootFolder $rootFolder,
61-
IUserManager $userManager,
62-
IPreview $previewGenerator,
63-
IConfig $config,
64-
IDBConnection $connection,
65-
IManager $encryptionManager,
66-
ITimeFactory $time,
67-
NoMediaService $noMediaService,
68-
SizeHelper $sizeHelper) {
19+
public function __construct(
20+
private readonly PreGenerateService $preGenerateService,
21+
) {
6922
parent::__construct();
70-
71-
$this->appName = $appName;
72-
$this->userManager = $userManager;
73-
$this->rootFolder = $rootFolder;
74-
$this->previewGenerator = $previewGenerator;
75-
$this->config = $config;
76-
$this->connection = $connection;
77-
$this->encryptionManager = $encryptionManager;
78-
$this->time = $time;
79-
$this->noMediaService = $noMediaService;
80-
$this->sizeHelper = $sizeHelper;
8123
}
8224

8325
protected function configure(): void {
@@ -87,126 +29,10 @@ protected function configure(): void {
8729
}
8830

8931
protected function execute(InputInterface $input, OutputInterface $output): int {
90-
if ($this->encryptionManager->isEnabled()) {
91-
$output->writeln('Encryption is enabled. Aborted.');
92-
return 1;
93-
}
94-
95-
// Set timestamp output
96-
$formatter = new TimestampFormatter($this->config, $output->getFormatter());
97-
$output->setFormatter($formatter);
98-
$this->output = $output;
99-
100-
$this->specifications = $this->sizeHelper->generateSpecifications();
101-
if ($this->output->getVerbosity() > OutputInterface::VERBOSITY_VERY_VERBOSE) {
102-
$output->writeln('Specifications: ' . json_encode($this->specifications));
103-
}
104-
$this->startProcessing();
105-
106-
return 0;
107-
}
108-
109-
private function startProcessing(): void {
110-
while (true) {
111-
/*
112-
* Get and delete the row so that if preview generation fails for some reason the next
113-
* run can just continue. Wrap in transaction to make sure that one row is handled by
114-
* one process only.
115-
*/
116-
$row = $this->atomic(function () {
117-
$qb = $this->connection->getQueryBuilder();
118-
$qb->select('*')
119-
->from('preview_generation')
120-
->orderBy('id')
121-
->setMaxResults(1);
122-
$result = $qb->executeQuery();
123-
$row = $result->fetch();
124-
$result->closeCursor();
125-
126-
if (!$row) {
127-
return null;
128-
}
129-
130-
$qb = $this->connection->getQueryBuilder();
131-
$qb->delete('preview_generation')
132-
->where($qb->expr()->eq('id', $qb->createNamedParameter($row['id'])));
133-
$qb->executeStatement();
134-
135-
return $row;
136-
}, $this->connection);
137-
138-
139-
if (!$row) {
140-
break;
141-
}
142-
143-
$this->processRow($row);
144-
}
145-
}
146-
147-
private function processRow($row): void {
148-
//Get user
149-
$user = $this->userManager->get($row['uid']);
150-
151-
if ($user === null) {
152-
return;
153-
}
154-
155-
\OC_Util::tearDownFS();
156-
\OC_Util::setupFS($row['uid']);
157-
15832
try {
159-
$userFolder = $this->rootFolder->getUserFolder($user->getUID());
160-
$userRoot = $userFolder->getParent();
161-
} catch (NotFoundException $e) {
162-
return;
163-
}
164-
165-
//Get node
166-
$nodes = $userRoot->getById($row['file_id']);
167-
168-
if ($nodes === []) {
169-
return;
170-
}
171-
172-
$node = $nodes[0];
173-
if ($node instanceof File) {
174-
$this->processFile($node);
175-
}
176-
}
177-
178-
private function processFile(File $file): void {
179-
$absPath = ltrim($file->getPath(), '/');
180-
$pathComponents = explode('/', $absPath);
181-
if (isset($pathComponents[1]) && $pathComponents[1] === 'files_trashbin') {
182-
return;
183-
}
184-
185-
if ($this->noMediaService->hasNoMediaFile($file)) {
186-
return;
187-
}
188-
189-
if ($this->previewGenerator->isMimeSupported($file->getMimeType())) {
190-
if ($this->output->getVerbosity() > OutputInterface::VERBOSITY_VERBOSE) {
191-
$this->output->writeln('Generating previews for ' . $file->getPath());
192-
}
193-
194-
try {
195-
$this->previewGenerator->generatePreviews($file, $this->specifications);
196-
} catch (NotFoundException $e) {
197-
// Maybe log that previews could not be generated?
198-
} catch (\InvalidArgumentException|GenericFileException $e) {
199-
$class = $e::class;
200-
$error = $e->getMessage();
201-
$this->output->writeln("<error>{$class}: {$error}</error>");
202-
} catch (DbalException $e) {
203-
// Since the introduction of the oc_previews table, preview duplication caused by
204-
// duplicated specifications will cause a UniqueConstraintViolationException. We can
205-
// simply ignore this exception here and carry on.
206-
if ($e->getReason() !== Exception::REASON_UNIQUE_CONSTRAINT_VIOLATION) {
207-
throw $e;
208-
}
209-
}
33+
$this->preGenerateService->preGenerate($output);
34+
} catch (EncryptionEnabledException $e) {
35+
$output->writeln('<error>Encryption is enabled. Aborted.</error>');
21036
}
21137
}
21238
}

0 commit comments

Comments
 (0)