Hot Folder Data Importing

With hot folder data importing, CSV files are imported automatically when they are moved to a folder that is scanned periodically by the system. 

acceleratorservices extension template comes with a batch package that enables automated importing of data from hot folders. 

The infrastructure enables the import of CSV files that are internally translated into multi-threaded ImpEx scripts. 

The infrastructure uses Spring integration to provide a service-based design.

Diagram of Components

The classes are structured into three major parts:

• Tasks executed by the Spring integration infrastructure

HeaderSetupTask
BatchHeader
HeaderTask
HeaderInitTask
ImpexTransformerTask
ImpexRunnerTask
CleanupTask

• Converters providing the ImpEx header and converting CSV rows into ImpEx rows with optional filtering

ImpexConverter
ImpexRowFilter

• Helper and utility classes

SequenceIdParser
RegexParser
CleanupHelper

General Flow

• Spring integration periodically scans the configured input directory for new files.
• If new files are found, they are moved to the processing subdirectory and then sent to the Batch Import pipeline, which consists of the following tasks:

HeaderSetupTask
HeaderInitTask
ImpexTransformerTask
ImpexRunnerTask
CleanupTask
ErrorHandler

• HeaderSetupTask: Creates a new BatchHeader.
• HeaderInitTask: Retrieves a sequence ID and (optionally) a language from the file name.
• ImpexTransformerTask: Creates one or many ImpEx files from the CSV input and writes error lines to the error subdirectory.
• ImpexRunnerTask: Processes all ImpEx files sequentially with multiple threads.
• CleanupTask: Deletes all transformed files and moves the imported file with an optionally appended timestamp to the archive subdirectory.
• ErrorHandler: Deletes all transformed files and moves the imported file with an optionally appended timestamp to the error subdirectory.

Configuration files

• hot-folder-spring.xml (<HYBRIS_BIN_DIR>/ext-accelerator/acceleratorservices/resources/acceleratorservices/integration)
• hot-folder-common-spring.xml (<HYBRIS_BIN_DIR>/ext-template/yacceleratorcore/resources/yacceleratorcore/integration)
• hot-folder-store-electronics-spring.xml (<HYBRIS_BIN_DIR>/ext-template/yacceleratorcore/resources/yacceleratorcore/integration)
• hot-folder-store-apparel-spring.xml (<HYBRIS_BIN_DIR>/ext-template/yacceleratorcore/resources/yacceleratorcore/integration)
• hot-folder-store-powertools-spring.xml (<HYBRIS_BIN_DIR>/ext-template/yb2bacceleratorcore/resources/yb2bacceleratorcore/integration)
• project.properties (<HYBRIS_BIN_DIR>/ext-accelerator/acceleratorservices)

Spring Integration Configuration

• file:inbound-channel-adapter: Scans a directory in a configurable interval and sends files to a configured channel under the following conditions: 
• Only files matching a specified regular expression are retrieved (filename-regex). 
• Files are processed in the order defined by the FileOrderComparator, using the following priority rule: 
• If a priority is configured for the file prefix, it uses the specified priority. 
• For files with equal priority, the older file is processed first. 
• file:outbound-gateway: Moves a file to the processing subdirectory.
• int:service-activator: Activates a referenced bean when receiving a message on a configured channel. The bean response is again wrapped in a message and sent to the configured output channel.
• int:channel: Sets up a channel.

Technical Background

• Spring integration periodically scans the configured input directory for new files (hot-folder-store-apparel-spring.xml)

 

project.properties

 

 

• New files are moved to the processing subdirectory (delete-source-files=true: deletes the original source files after writing to the destination) and then HeaderSetupTask is called (method attribute: method of the referenced bean)

HeaderSetupTask

• catalog: The catalog to use. This setting is applied to the default header substitution: $CATALOG$
• net: The net setting to apply to prices. This setting is applied to the default header substitution: $NET$

• HeaderInitTask is called

 

HeaderInitTask

• sequenceIdParser: The regular expression used to extract the sequence ID from the file name.  
• languageParser: The regular expression used to extract the language from the file name. 
• fallbackLanguage: The language to use if the language is not set in the file name. 

• ImpexTransformerTask is called. init-method is executed first and then the specified method of the bean.

 

 

ImpexTransformerTask performs the following tasks: 

• It retrieves all configured converters matching the file name prefix.
• For every converter found, it converts the input file as follows: 
• It adds the ImpEx file header once with substitutions
• It converts all rows if they are not filtered
• If the line has missing input fields, it adds the line along with the error message to a file in the error subdirectory

ImpexTransformerTask

• fieldSeparator: The separator to use to read CSV files (default value is ,). 
• encoding: The file encoding to use (default value is UTF-8). 
• linesToSkip: The lines to skip in all CSV files (default value is 0).
• converterMap: Used to map file prefixes to one or multiple converters to produce ImpEx files.

CleanupHelper

• timeStampFormat: If set, appends a timestamp in the specified format to input files moved to the archive or error subdirectory.

ConverterMapping

Converter corresponding to base_product file name prefix

Converter

• header: The ImpEx header to use including header substitutions:
  $NET$: the net setting
  $CATALOG$: the catalog prefix
  $LANGUAGE$: the language setting
  $TYPE$: an optional type attribute that can be applied if filtering is configured 
• impexRow: The template for an ImpEx row adhering to the syntax:
  Syntax: {('+')? (<columnId> | 'S')}
  The '+' character adds a mandatory check to this column. Any lines with missing attributes are written to an error file in the error subdirectory.
  The 'S' can be used for writing the current sequence ID at the template position. Optionally, columns can be quoted by enclosing the column in the template with quotation marks.
• rowFilter: An optional row filter. The supplied expression must be a valid Groovy expression. The current row map consisting of column ID and value is referenced by row.
  Configuring multiple converters with row filters gives the option to split a supplied CSV input file into different ImpEx files according to specified filter criteria. 
• type: An optional type that can be retrieved in the header using the header substitution $TYPE$.

Converter corresponding to variant file name prefix

ImpexRowFilter

• ImpexRunnerTask is called

 

All generated ImpEx files are sent to the platform ImportService in the AbstractImpexRunnerTask sequentially.

• CleanupTask is called

 

The Type System

Type = type definition in items.xml + its Java implementation
Item = object instance of a Type

The Item type is the supertype of all types. 

Attributes of a Composed Type

• Can be a reference to
• a Composed Type
• a basic Java Type
• Can have a localized name and description
• Can have a default value

Creating Types

Adding New Attributes to a Type

Moving a Type (from one extension to another)
Move TypeA from ExtensionA to ExtensionB

extensiona-items.xml
<itemtype
    generate="true"
    code="TypeA"
    jaloclass="de.hybris.extensiona.jalo.TypeA"
    extends="GenericItem"
    autocreate="true" >

   <deployment table="mytype_deployment" typecode="12345"/>
   ...

</itemtype>

1. Change the package path of jaloclass 
2. Move the type definition to new items.xml file
3. Build -> Start -> Update

extensionb-items.xml
<itemtype
    generate="true"
    code="TypeA"
    jaloclass="de.hybris.extensionb.jalo.TypeA"
    extends="GenericItem"
    autocreate="true" >

   <deployment table="mytype_deployment" typecode="12345"/>
   ...

</itemtype>

Limitations

• Not allowed to change the deployment typecode
• Cannot move if it affects the classpath

Cannot move TypeA from ExtensionA to ExtensionB.
Compilation fails in this case because the ExtensionC still needs TypeA that is used to extend TypeC.
Possible workaround would be to relate ExtensionC with ExtensionB, but it may not fit your business objectives.

Available Types
core-items.xml (/platform/ext/core/resources)

1. AtomicTypes
2. CollectionTypes
3. EnumerationTypes
4. MapTypes
5. RelationTypes
6. ItemTypes

AtomicTypes

Does not have a code attribute, instead class attribute is used as its reference

To localize atomic types, add the following in locales_xx.properties (resources/localization)
type.localized:<class attribute>.name=<value>
type.localized:<class attribute>.description=<value>

Create a new atomic type

• Create a class for the atomic type. Eg: PK.java
• The created class has to implement Serializable interface

• Define the class in items.xml file

CollectionTypes

code attribute - unique identifier
elementtype attribute - type of elements in the collection
type attribute – List (ordered items), Set (no duplicates), SortedSet (unique ordered items)

CollectionTypes vs. RelationTypes (Use RelationTypes whenever possible)

• Maximum length of a database field is limited and hence the value may get truncated
• As the database entry only contains the PKs, you cannot get more details on each PK. You would need to query again to get further details.

EnumerationTypes

Storage of localized values
Localized values are stored in a separate database table, 
whose name = name of the table the type is stored in + the suffix lp (localized property)
E.g.; If the type is stored in the table sampletype, then its localized values are stored in the table sampletypelp.
 
MapTypes

argumenttype attribute - Key
returntype attribute - Value

RelationTypes

Internally, the elements on both sides of the relation are linked together via instances of a helper type called LinkItem.
LinkItems hold two attributes, SourceItem and TargetItem that hold references to the respective items.

ItemTypes

ItemTypes/ComposedTypes hold meta information on types and the types' attributes and relations, including the item type's code (unique identifier), its JNDI deployment location, the database table the item will be stored in and the type's Java class. 

Every type may have any number of attributes. 

Every attribute that is inherited downwards has its settings stored separately for each children type. That way, it is possible to override attribute access rights inherited from the supertype for a child type, so that you may set an attribute to be writable for your self-defined types that wasn't set writable on the type the attribute was originally defined with.


Creating Items at runtime

Creating Items in the Extension's Manager
Every extension has a Manager that is responsible for item handling. By calling the Manager's creation methods and passing the necessary parameters for the new item as a Map, you can create items.

...
Map params = new HashMap();
params.put( HelloWorldWizardCronJob.SCREENTEXT, getScreenText() );
params.put( HelloWorldWizardCronJob.ACTIVATE, isActivate() );
params.put( HelloWorldWizardCronJob.INTERVAL, getInterval() );
params.put( HelloWorldWizardCronJob.CODE, "HelloWorldWizardCronJob" + String.valueOf( jobNum ) );
params.put( HelloWorldWizardCronJob.JOB, hwwj );
HelloWorldWizardCronJob hwwcj = HelloWorldWizardManager.getInstance().createHelloWorldWizardCronJob( params );
...

Creating Items Generically
To create a new instance of a certain type,

• Select the respective ComposedType and

ComposedType item = getSession().getTypeManager().getComposedType("mySampleItem");

• Call its newInstance() method

item.newInstance(ctx, params);

Pass the necessary parameters for the new item as a Map

initial modifier

To set an attribute to be initial: Set the initial modifier in items.xml to true

<attributes>
   <attribute qualifier="code" type="java.lang.String">
      <modifiers initial="true"/>
      <persistence type="property"/>
   </attribute>
</attributes>

Checking the Mandatory Item Attributes
Inside the createItem() method, we can use checkMandatoryAttribute() method to check if each mandatory attribute has a value.

checkMandatoryAttribute(MyType.MYATTRIBUTE1, allAttributes, missing, true);
checkMandatoryAttribute(MyType.MYATTRIBUTE2, allAttributes, missing, false);

This method has four parameters:

• String qualifier - Reference to the attribute to check
• ItemAttributeMap allAttributes - Map with the initial attribute values
• Set missingSet - Set which accepts references to all attributes for which no value has been set
• Boolean nullAllowed - Specifies whether a null value in the ItemAttributeMap is written to the item as a null value (true) or whether the null value is treated as a missing value (false). Defaults to false.

Relations

Redeclaring 1:n Relations (redeclare=true)

Define a relation on an abstract level (between two abstract classes), and then re-declare this relation in concrete classes to point to concrete subclasses.

Steps:

• Define a relation between two abstract classes

• Redeclare the definition of Order item such that it contains only OrderEntries

• Add OrderEntry CollectionType

• Redeclare the definition of OrderEntry item

Custom Ordering

Custom property: ordering.attribute
To specify which attribute will be used to order the many-side items when retrieving from the database. 

Defined the many-side as ordered=false. There is no need for the ORM to add an additional ordering column, therefore the many side is ordered=false. 


Condition Query

Custom property: condition.query
Holds a string that is added to the where part of the select query generated for a one-to-many or many-to-one relation.

• Shouldn't contain any order by part as it is added at the end of a generated query.
• Only for relations of a one-to-many or many-to-one type.
• Can only be defined in one end of the relation. Must be defined in either sourceElement or targetElement that have the many cardinality.