As mentioned in 2024.1 1. The overall design of the system, the Import process is a two-step process. The middle part is the Queue. Let's take a closer look at how it works.

Fetch records from NetSuite

Each process starts in the netsuite:cron command. When run mode is either import or importToQueue this workflow is triggered. Let's take a closer look on how this works.

The main method that is called is \RocketWeb\NetSuite\Model\Process::processImport(). The same method is called in both run modes, only when the parameter $processQueue is true (run mode = import) which calls \RocketWeb\NetSuite\Model\Process::processImportQueue() before it runs the rest of the logic. The processImportQueue() is discussed in the next section.

Get the list of Importable Entities

Going forward through the call, we fetch the current timestamp $time from NetSuite server and process the last time this process ran (\RocketWeb\NetSuite\Model\Process::getUpdatedFromDateInNetsuiteFormat()). We fetch the list of importable entities (\RocketWeb\NetSuite\Model\Process\ImportProcessor::getImportableEntities()) that are filled thru di.xml in each of the modules.

Example

    <type name="RocketWeb\NetSuite\Model\Process\ImportProcessor">
        <arguments>
            <argument name="processors" xsi:type="array">
                <item name="inventoryitem" xsi:type="object">RocketWeb\NetSuiteProduct\Model\Process\Import\Item</item>
            </argument>
        </arguments>
    </type>

Customization

This allows extending the list of importable entities from NetSuite to other Entities or Records. Create your own Import process (which should extend \RocketWeb\NetSuite\Model\Process\Import\AbstractImportProcessor class) and add logic to process records fetch from NetSuite.

Loop Over All Importable Entites

The next part of the code is simple - we loop through all Importable Entity Processors gathered. If Processor is not active (\RocketWeb\NetSuite\Model\Process\Import\AbstractImportProcessor::isActive()), we skip it.

Fetch Entity Records from NetSuite and process them

We fetch the records from NetSuite using \RocketWeb\NetSuite\Model\Process\Import\AbstractImportProcessor::queryNetsuite() method. The method queryNetsuite() is quite simple - in the first round, it creates a Search request which is then used to create SearchMoreWithId requests for all other rounds. It returns an array $records which is then processed further based on the Importable Entity Type. If Type is \NetSuite\Classes\RecordType::inventoryItem or \NetSuite\Classes\RecordType::assemblyItem then we just save the $records fetched from NetSuite into Queue table. For all other entities additional check \RocketWeb\NetSuite\Model\Process\Import\AbstractImportProcessor::isMagentoImportable() and isAlreadyImported() is ran. Once all entity IDs are fetched we use different call \RocketWeb\NetSuite\Model\NetSuite\ServiceRepository::fetchMultipleRecordsFromNetSuite() to receive full records with all information (which doesn't happen with queryNetsuite() call). Those records are then saved into Queue table.

Post-process steps

The last part of the code takes care of saving the $time fetched at the beginning of the method into DB. We are using the Magento flag manager to save the timestamp of when the process was run last time. For this part of the process, we are using last_import_queue_run_date flag.

Process Import Queue

Each process starts in the netsuite:cron command. When the run mode is either import or processQueue this workflow is triggered. Let's take a closer look on how this works.

The main method that is called is \RocketWeb\NetSuite\Model\Process::processImportQueue(). The same method is called in both run modes.

Fetch Batch of Queue Messages and Process them

The batch size is defined based on configuration value. The queue batch is then fetched. For modification purposes, we have an empty method \RocketWeb\NetSuite\Model\Process::beforeProcessImportBatchData() which is a hook for customization plugins. An example of plugin usage of that hook can be found in \RocketWeb\NetSuiteProduct\Plugin\ImportQueuePrefetchDataPlugin::afterBeforeProcessImportBatchData() where we prefetch products when we are processing \NetSuite\Classes\RecordType::inventoryItem messages.

Loop over Messages and Process them

Inside \RocketWeb\NetSuite\Model\Process::importMessages() we loop over the $messages and unpack them. Based on the $message action type, the Processor $processModel is fetched. Before we run the Processor, an event netsuite_import_item_process_before is triggered which allows easy modification of $message data. The rest of the code is simple - we run the \RocketWeb\NetSuite\Model\Process\Import\AbstractImportProcessor::process() which takes care of actual processing of the $message to be ready for Magento import. The result of the process() is then added into \RocketWeb\NetSuite\Model\ImportQueueManager::import() which takes care of keeping the data ready for import in a single place based on Entity Type.

This section will be updated with a list of all Processors and documentation on what they do.

Batch import

Once all of the batches are processed, the last step is to import the data that is ready. This happens in \RocketWeb\NetSuite\Model\ImportQueueManager::commit() where we loop over data and fetch the correct Entity Processor $importProcessor (\RocketWeb\NetSuite\Model\ImportQueueManager::getImportProcessor()) and pass it for processing \RocketWeb\NetSuite\Model\ImportQueue\EntityProcessor::process(). This method triggeres \RocketWeb\NetSuite\Model\Importer::importData() which runs \Magento\ImportExport\Model\Import class to handle the final validation & import of the data. The rest of the \RocketWeb\NetSuite\Model\ImportQueueManager::commit() method takes care of \RocketWeb\NetSuite\Model\ImportQueue\EntityProcessor::postProcess() call and reindexing where needed.

The \RocketWeb\NetSuite\Model\ImportQueueManager::commit() is a great place for xdebug to stop execution. It allows the developer to see what is the actual data that is being prepared for import into Magento. For specific Entity debug, dig into a specific Processor.

Customization

This part of the process usually doesn't need any modifications. If you created a new Entity to be processed in the previous section, you need to make sure that the correct EntityProcessor is used (or a new one created if needed).

Browser not supported