How to overcome hurdles when transferring data in a PIM project

Practical recommendations for a smart approach to data transfer

When introducing a new PIM system, the biggest hurdle is data transfer. If you take a clever approach, you can make your work much easier. I would like to outline the term "data transfer" in a little more detail: this should cover both the one-off, initial data transfer and the ongoing interfaces to upstream systems, in particular the ERP system.

The initial data transfer for initial filling can be carried out from an existing PIM system - this is known as data migration - or on a "greenfield site". Even on a "greenfield site", if there is no PIM system or similar in place, you will find databases, whether in Excel files, in the CMS or in documents. This data, plus the media files (images, documents, videos), must be prepared for import into the (new) PIM system. And there are of course - and rightly so - fears that the effort involved will be too much. I want to allay these fears to some extent by showing you ways to simplify and speed up the data transfer process. However, there is of course still work to be done, even if you proceed as cleverly as possible.

Use existing data sources such as the ERP system

My first tip is to continuously obtain as much existing data as possible from the ERP system via an interface. Of course, only the data for which the ERP system has data sovereignty. Incidentally, it is very important to define the data sovereignty for each piece of information: Where should the information originally be maintained? In the ERP system, in the PIM system, etc. This information should then only be maintained in this system and then accessed. The advantage is that you don't have to prepare and transfer data that you can access on an ongoing basis, which saves a lot of work. You "only" have to make sure that the data from the ERP system "fits" (which is unfortunately not always the case). Of course, the above is self-evident - and yet I almost always find that not all PIM-relevant data is accessed in the ERP system.

Procedure for data transfer

Let's move on to the second tip. There are basically two completely different approaches to data transfer: On the one hand, the procedure of completely transferring all data in one go and, on the other, the step-by-step procedure. Step-by-step means that you start with a specific product area and carry out all the derivations for this (MVP = Minimal Viable Product), for example providing the data for the website or store or producing the corresponding catalog pages, data sheets etc.. Both approaches have advantages and disadvantages. The main advantage of the step-by-step approach is that you achieve success quickly and are motivated as a result. Progress is visible to everyone involved and is not a "black box". The main advantage of complete data transfer is that you have an overview of all the data as a whole and are therefore more efficient and can also recognize structural data problems more systematically. Both approaches have their justification; the choice should in any case suit those involved.

Technical formats for data transfer

The third tip is a little more technical. In what form should the data be prepared? This often involves Excel, BMEcat, XML and JSON. BMEcat, XML and JSON may be used for an interface, but I would clearly recommend Excel for the data preparation phase. For the simple reason that you can adapt and supplement the data there. The other formats are good for the machine, but not for humans. Of course, Excel is not just Excel: it depends on how the data can be prepared, column by column or row by row. Column by column means that the field names are in the columns and the content is in the rows by product. Although this leads to wide Excel lists, the overview is much better than if the field names are in the rows. I can only recommend that you take a look at the import options, because the additional effort due to functional deficiencies can quickly run into weeks. Incidentally, the best thing is if the import files for Excel are based on a template (XML schema XSD) and validations and conversions to XML are therefore possible.

Checklist for data preparation

So far we have talked about tips on how to proceed, now I would like to move on to the data content itself. This will determine how useful the data is in the long term. This is where the details really count and you should aim for 100% data quality, so be sure to choose thoroughness over speed, because once the data is in use, corrections become all the more time-consuming!

You may think when reading the checklist, oh how petty, but all the tips have a practical background (from over 100 PIM projects) and all the shortcomings will catch up with you at some point.

Here is my checklist for the data preparation of attributes:

• Assign unique database names carefully
  My recommendation is to use "natural" names instead of consecutively assigned numbers (short, unique and comprehensible).

• Carefully assign designations in the required languages
  Also assign alternative designations in one go, for example abbreviated column names for table column or drawing labels, explanations, etc.

• Use classification
  Cluster attributes into classes so that they are suitable for maintaining similar products

• Specify selection lists with fixed entries - if at all possible - to enforce uniformity
  It should also be possible to use sequences that deviate from alphabetical sorting.

• Assign data types correctly
  Use the number data type (if they are used for conversion and are to be formatted as numbers). EAN, customs tariff number etc. should be saved as text and not as a number.

• In the case of numbers: Separation from the units
  
  
• Splitting min-max attributes into two attributes
  

• Mark language-neutral content as such
  To avoid unnecessary translation work

• Specify input checks
  For example, number range, text length, number of decimal places, uniqueness, multiple or simple assignment

• Assign authorizations
  For example, "Block data maintenance" for content from the ERP system

• Mandatory fields: Specify for checks whether the attribute is a mandatory or optional field in the context of a class.
  
  

The following PIM functionalities are useful for managing the attributes:

• Number formatting: Country-specific decimal and thousands separators
  Setting options for rounding rules and number of decimal places. If, in exceptional cases, texts contain numbers, the numbers must also be formatted ("pseudo-numeric").

• Depict dependent attributes as such (multidimensional) in order to preserve the context
  For example, nominal voltage as a function of frequency

• Concatenate attributes
  For example, concatenate min-max attributes or attributes with their units

• Convert attributes into formulas
  For example, convert metric measurements into imperial measurements. Map complex calculations with conditions as scripts

• Insert attributes in texts or graphics as placeholders
  To ensure consistency in all content

• Assign images, for example icons
  Because images are often used for visualization in catalogs and websites instead of textual designations

• Inheritance concept
  To be able to maintain attributes not only at item level, but also at product group level for all items contained therein, etc.

Use of AI for data transfer

Finally, I would like to share my experience with AI support. I believe that AI can develop its full potential if the basic data is of high quality.

I would like to give two examples to illustrate this:

• In a pilot project, the classification of products into the standard ECLASS classifications is being carried out with AI support. The existing, very cleanly maintained data is being used for this. I am sure that without this database, the AI support would not be successful.

• In customer projects, descriptive and advertising texts are generated for marketing. The tidier the basic data is, the better the AI support works, for example with ChatGPT.

Can AI also be used for initial data preparation? Hmm, theoretically yes, the only question is whether it pays off! Typical times for data preparation are several months on average. So you can possibly save weeks with AI. On the other hand, you have to bear in mind that it requires a correction run, and corrections are often more time-consuming than the initial manual preparation. I would therefore recommend doing the initial data preparation manually. And then, as a freestyle, use the AI on a proper database for other purposes and let it do its magic.

Conclusion

Think about the data transfer procedure before you start:
Define the data sovereignty in conjunction with the ERP system. Decide on a step-by-step or complete procedure. Decide which tools and formats are to be used. Go through the checklist for data preparation and compare the requirements with the experts and with the functions of the PIM system. Above all: benefit from these tips and feel free to give me feedback!