File locking should be configurable

[mfendeksilverstripe]

File locking should be configurable

DefaultQueueHandler relies on file based locks for the immediate queue. File locks are files placed under folder configured in
cache_dir (QueuedJobService). This folder is placed under the temp folder which doesn't work really well with multi instance environments and parallel queue runner.

I suggest to allow this feature to be disabled by assigning null to the cache_dir configuration field.

Context

public function activateOnQueue()
{
    // if it's an immediate job, lets cache it to disk to be picked up later
    if ($this->JobType == QueuedJob::IMMEDIATE
        && !Config::inst()->get(QueuedJobService::class, 'use_shutdown_function')
    ) {
        touch($this->getJobDir() . '/queuedjob-' . $this->ID);
    }
}

protected function getJobDir()
{
    // make sure our temp dir is in place. This is what will be inotify watched
    $jobDir = Config::inst()->get(QueuedJobService::class, 'cache_dir');
    if ($jobDir[0] !== '/') {
        $jobDir = TEMP_FOLDER . '/' . $jobDir;
    }

    if (!is_dir($jobDir)) {
        Filesystem::makeFolder($jobDir);
    }
    return $jobDir;
}

public function cleanupJob()
{
    // remove the job's temp file if it exists
    $tmpFile = $this->getJobDir() . '/queuedjob-' . $this->ID;
    if (file_exists($tmpFile)) {
        unlink($tmpFile);
    }
}

[michalkleiner]

What other type of locking mechanism it would use otherwise if the file one wasn't available? Would it fallback to e.g. dynamodb or session or the likes?

[mfendeksilverstripe]

@michalkleiner As fas as I can see this file locking mechanism is used only for immediate queue and only in case the use_shutdown_function option is disabled. TBH I'm not even sure if this feature is needed at all. Maybe this is some legacy code? As far as I know any mutex related functionality is relying on DB based locking mechanism and this is used for all queues.

[michalkleiner]

So what problem is this solving (apart from making it configurable)? Is the immediate queue only used when running in dev mode where it could be fairly assumed people don't develop in multi-server environments?

[mfendeksilverstripe]

I'll do some local testing with different project setups and capture the relevant information in here.