typo3 - In an extbase extension, how to access the persistence layer from a scheduler task?

Question

What sounds a bit academic in the title is actually quite straightforward: I have set up a TYPO3 6.1 extbase extension that I've equipped with a scheduler task. The task is supposed to import a CSV file and save it into the extension's database fields.

But how do I tell the scheduler task to use the extension's model etc. and save the received data into the persistence layer?

I've seen this answer to a similar question: Execute repository functions in scheduler task and I think it points the right way, but I think need a full example to start understanding how the dependency injection works.

Answer 1

First you have to consider the aspect of performance:

If you want to insert a big amount of data, you should not use the Extbase persistence for such a task. Because if you do so, it will generate an object for each row you want to insert and persist it immediately. This is quite slow and has a big memory footprint.

If you don't have much data or you split the jobs (e.g. perform 100 import jobs per scheduler run), then use the Extbase persistence.

You can have both in CommandController context, and since CommandControllers are straight-forward to set up, you should go for them instead of an own Scheduler task.

Using Extbase persistence

In the CommandController, inject your repository:

/**
 * myRepository
 *
 * @var VenorMyextDomainRepositoryMyRepository
 * @inject
 */
protected $myRepository

Then iterate through the rows you want to import (foreach) and create a new object for every row and add it to your repository:

 $myObject = $this->objectManager->get('VendorMyextDomainModelMyModel');
 $myObject->setProperty('foo');
 $myObject->setOtherProperty('bar');
 $this->myRepository->add($myObject);

To actually save the objects to the database, you need to persist them. So you also inject the persistenceManager:

/**
 * @var TYPO3CMSExtbasePersistenceGenericPersistenceManager
 * @inject
 */
protected $persistenceManager;

And use it:

 $this->persistenceManager->persistAll();

You should not do that for every single object (for performance reasons); but for memory usage reasons you should not wait until after thousands of objects to persist, either. So you just insert an iterator to your foreach loop and persist every 20th, 40th, or whatever loop.

Please don't forget that the Scheduler works in Backend context, so the TypoScript must be available by module.tx_yourext. If you want to share the same settings/storagePid with the frontend part of your app, use

module.tx_yourext.persistence < plugin.tx_yourext.persistence
[...]

The TypoScript needs to be present in the root page of your website for backend modules/CommandControllers to use them. I suggest you add the stuff to myext/Configuration/TypoScript/setup.txt and add the static template of your extension to the root page.

Using DataHandler

The TYPO3 DataHandler (formerly TCEmain) is the engine the TYPO3 backend uses for inserting and modifying database records. It is very powerful.

Instead of an object, inside your loop you create an array containing all the data. The first array index is the table, the next level is the affected record, where NEW means that a new record is created. Then you can just set every field of a table with the desired value

    $data = array();
    $data['be_users']['NEW'] = array(
        'pid' => 0,
        'username' => $staffMember['IDPerson'],
        'password' => md5(GeneralUtility::generateRandomBytes(40)), // random password
        'usergroup' => '1,2',
        'email' => $staffMember['Email'],
        'realName' => $staffMember['Nachname'] . ' ' . $staffMember['Vorname'],
        'lang' => 'de',
    );

Now you can make an Instance of DataHandler and persist the changes:

    /** @var $tce t3lib_TCEmain */
    $tce = GeneralUtility::makeInstance('TYPO3CMSCoreDataHandlingDataHandler');
    $tce->bypassAccessCheckForRecords = TRUE;
    $tce->start($data, array());
    $tce->admin = TRUE;
    $tce->process_datamap();
    $newRecordsUidArray = $tce->substNEWwithIDs['NEW'];

Please note the line $tce->admin = TRUE. This suggests to DataHandler that an admin is performing the action. This is convenient because you don't have to set allowed exclude fields for the Scheduler user and can also insert records to PID 0. But it is a possible security flaw, so carefully consider its usage.

Records inserted/updated by DataHandler logged correctly, can be reverted etc.. You can find some examples (such as adding pictures, resolving MM relations) here. In this case all DataHandler related functions were moved to an external repository class that is injected to the CommandController as described above (it's just named in Extbase convention).

A good overview of DataHandler functions can be found here.