The more product data, the more work for developers, the heavier your Data Layer. Tealium’s “Dynamic Data Layer Management” promises to ban at least the developer part forever. The idea is to enrich your Data Layer in real time with data (like “best-price” labels) that is hosted somewhere outside on a CDN. It has huge potential and is something you cannot do with a other Tag Management Systems. But does DDLM live up to its promise? We tested it.

Ever more and more interesting information is being added to your site all the time. This week, a new feature comes out where the user can see if the price of a product has dropped in the last week, next week he can see a custom product label like “5-star eco ratings”, and so on… If you want any of that information to be available for easy consumption and analysis in your Adobe or Google Analytics reports, that is often impossible without Data Layer enhancements, i.e. more developer work.

Don’t Confuse Tealium’s Two “Data Layer Enrichment” Features

Tealium, creator of one of the first Tag Management Systems called “Tealium iQ“, must have observed this problem, which is maybe why last year they came out with a promising feature. Tealium calls it “Dynamic Data Layer Management” (on their Tealium iQ page) or “Data Layer Enrichment” (in their documentation). The latter shouldn’t be confused with the “Data Layer Enrichment” performed by Tealium’s “Audience Stream“. There, Data Layer Enrichment (DLE) is a component of Audience Stream’s “Visitor Stitching” and allows you to send additional user data from the user’s history (like “devices used”) or from other systems, e.g. the age group from your CRM, into the Data Layer. So the data becomes available in the current user’s session as soon as the user has provided at least one form of identification (e.g. his login ID) and can be used for tracking or targeting.

Since I want to avoid confusion with that Audience Stream feature, I will refer to the tool we are talking about here as “Dynamic Data Layer Management” (DDLM). What’s cool in the first place is that DDLM is free for you if you are a Tealium iQ customer. So we (Swiss marketplace siroop.ch) went out to test it!

How Does DDLM Work?

Each website usually has a database with all the data presented to the visitor (and the data not presented to the user, but known to the website owner). That is usually the product database (the most essential part of a shop) in a Product Information Management (PIM) tool, or an article database in a CMS.

Whenever the Web Analyst wants to track a new piece of information (like, the author of an article or the stock availability, the user ratings score, the size of the products, their color etc.), in a traditional world she would ask the developer to add that piece into the Data Layer as a new product-scoped variable (e.g. prod_rating: [“4.5”]) so the TMS can pick it up and send it to any tracking or marketing tool that can make use of it.

Instead of Yet Another Data Layer Variable: Plug the Data Layer Into the Database

Now, having the developer provide all that information in the Data Layer requires the usual release cycles, waiting and communication time, and it takes away development resources from more important tasks, so it quickly gets to a point where you think twice whether you want to bother the dev team with yet another Data Layer variable (also because of the other downsides of “yet another Data Layer variable” mentioned in Part 1).

Here comes DDLM to the rescue! It is designed to be plugged into your database directly or indirectly. When everything is up and running, DDLM then inserts the necessary product or article information into your Data Layer on the fly in real-time and you can use it like normal data layer variables. And your developer can do more important things.

The Steps To Make It Work:

1. Define the level of granularity of your product data (the “Lookup Variable”). Usually, there is not just one Product ID level in your database, but several IDs depending on the information hierarchy of the product data. Some information is only available on the lower levels. As an example, siroop has the following product information hierarchies, each with its own ID:
   • Product Group IDs (e.g. all iPhone 7’s, all adidas Swift Run sneakers): Here we have only lowly granular data like the product category and brand.
   • Product Abstract IDs (all red iPhone 7’s, green adidas Swift Run sneakers): More product features become available. This level is also referred to as “imaging” or “image-giving”, because from this level on, a product has enough attributes that it can be shown on a picture (a red iPhone looks the same no matter how many GBs are inside, but a blue iPhone looks different from a red iPhone, so it has a different “Abstract” ID)
   • Product Concrete IDs (red iPhone 7’s with 64 GB, green adidas Swift Run sneakers of size 45): The most granular level in terms of product features. All products with the same Concrete ID are identical. No matter which merchant offers this product, it is always a red iPhone 7 with 64GB (and not one with 128 GB for example).
   • Merchant Product IDs (red iPhone 7’s with 64 GB offered by Merchant Fust, adidas Swift Run sneakers of size 45 offered by merchant siroop AG): Context Features of the product  like the actual price and stock availability as well as a link to Merchant’s attributes like delivery time, shipping costs (every merchant may have different values here)I am going into more detail here as this is a point where many E-Commerce Marketing tools fail to recognize the complex reality of E-Commerce. Most, e.g. Google Analytics, only know one level of Product IDs. In Adobe, it is basically the same, but you can work around it (see part 1).

     Tealium’s Dynamic Data Layer Management allows various various lookup variables and thus different levels as long as the same ID string does not appear on different levels (e.g. if you have a Concrete ID 12345 and an Abstract ID 12345, it won’t work).

     But it is much easier to start with one level, so you should analyze first on which level you will find the biggest benefit for your tracking/reporting/retargeting purposes. A higher level means less data, but it also means less issues with scalability like row limits in reports (see part 1) or upload limits for Tealium’s API. So don’t go for the most granular level just for the sake of getting everything in its ultimate granularity if you do not need it (even if everyone around you screams “we need the rawest of raw data possible!”). At siroop for example, we use the Concrete ID level for example even if some attributes are only available on the Merchant Product ID level – we have good workarounds for those.

     Needless to say, the ID for the level you choose needs to be in your Data Layer on the site already. If it is not there, there is no way for Tealium to map it. So if the developer has to implement at least one thing, it is this ID.

2. Create a feed that contains the chosen Product ID plus the data that you want to have inserted into your Data Layer in real-time. At siroop, we simply created this feed in ProductsUp from the general product data feed we already have in use for Google Shopping and other Marketing/Analytics purposes like Adobe classifications (see part 1). The good thing that this feed contains more than just data from the product database, you can also feed data from third-party sources in there, e.g. price comparison engines that tell you if your product is the cheapest at the moment or a list of currently advertised products created by your Category Managers. The feed should be continuously updated, best is several times a day, so you can have the most current product data ready at all times. ProductsUp usually regenerates all of its feeds about every other hour.
3. Upload (via HTTP POST calls) the data from your product feed to Tealium’s Content Delivery Network (CDN) via its Data Layer API. For every product ID, you need a separate call. Coordinate with Tealium the first time you do it, as you may break their API limits if you are posting too many products per second or too many IDs in total. Here you need this documentation and one-time help from a developer or your agency. We did it with the help of Feld M, one of our agencies.Make sure you post only the delta; i.e. only those products whose data has changed since the last upload. Posting tens or hundreds of thousands of product IDs several times a day will bring Tealium’s API down on its knees. ProductsUp can help you with creating a “delta-only” feed, or your developers can do a diff with the last feed data before posting it to Tealium’s API. In the end, Tealium will have a file for each of your products on their CDNs, e.g. https://tags.tiqcdn.com/dle/eos.commerce.ag/main/890474-1.js
   
   The file ends on {{Product ID]].js, e.g. “890474-1.js” (890474-1 is the Concrete Product ID of an iPhone 7 32 GB). It contains product data for this ID. This product data will then be retrieved by Tealium iQ in real-time when the user views the Product Page.
4. Add and configure the DDLM Extension in Tealium iQ (where it is called “Data Layer Enrichment”). Put it as high as possible in your Extension order (we have it as the second “All Tags” Extension).
   Choose one or more Lookup variables from those in your existing Data Layer, e.g. “prod_id”. Important: The Extension only supports mapping one ID per page view, so the variable cannot be an array! That is one of its biggest weaknesses of the extension, as on siroop.ch as well as most other E-Commerce sites, a user can interact with multiple product IDs at once (e.g. when he has several products in the cart). So this is why we need another extension that runs before the DDLM Extension. That other extension takes the first Concrete Product ID out of the array with Concrete IDs in the page’s Data Layer and writes that ID into the Lookup variable. For example,

   prodConcreteIds = ["1234-1", "2345-1", "3456-1"] // array of all product IDs on the page

   results in

   prodDDLM_LookupId = "1234-1" // lookup variable

   Set “Overwrite on merge” only to “Yes” if you want the Data Layer Extension to overwrite Data Layer variables with the same names as those on the Tealium CDN (e.g. when you think the shop is producing wrong or less recent data and the data you post to the Tealium Data Layer API is more correct). Usually you should use this feature to add variables and not override existing ones, so we set it to “no” to avoid any unpredictable side effects.

Now You Publish and You’re Done, Right?

That’s what we hoped. But not so fast. It took us a while to get the DDLM working. And every time it seemed like we had finally made it, a new technical issue came up.

First, there was the 300,000 product ID limit which made the feature useless for a shop with a million products like ours. After a while, Tealium lifted the limit. Then we kept running into API limits or API bugs. They were subsequently fixed by Tealium, but we felt a bit like guineapigs trying out a bit of an unfinished product.

On our side of things, we underestimated the pace of change in the product data which in turn overtaxed our data uploader (at some point, we were waiting for a week until all the products had been uploaded… ;)).

Your Extensions Will Now Run Twice!

Then there was the “noview” flag that the DDLM extension sets. That means it changes the logic of how all your extensions run in Tealium iQ. The flag makes Tealium execute all its extensions once, then pause until the DDLM extension has received its data and then run them all again.

This sounds trivial, but it lead to severe tracking issues and required numerous changes to the rest of our Tealium setup during weeks of unpleasant surprises and emergency work. Now even two months after activating, we still find data that is broken or completely lost due to this. This is certainly the most annoying part of the DLE.

Messed Up Logic and a 100% Repeat Purchase Rate

A lot of logic got messed up due to this: Some values were all of a sudden incremented twice because the extension to increment values now ran twice. Another example was an extension that first checks a cookie that holds the information if someone has bought already.

If “yes”, it sets a “repeat order = yes” counter when the customer buys. If “no”, it sets the cookie to “yes”, so the next purchase will be counted as a repeat purchase. But now, if this extension runs twice, the second time the cookie check is done, this cookie is already there because it was created during the first run. So all orders were suddenly being counted as repeat orders which led to a 100% Repeat Purchase rate in our reports and dashboards…

The reason is that the DDLM sets the utag.cfg['noview'] flag to ‘true’ whenever there is a Product ID in your Data Layer. This flag is commonly used for single-page applications where the application wants to control when the initial pageview is fired. If set to ‘true’, the “All Tags” Extensions run once, every Data Layer variable set during that run (the b object) is thrown away afterwards, then the Extensions run again and set the Data Layer variables again (and do everything else again which you programmed them to do, like setting and checking cookies). Then Tealium distributes the Data Layer data to the tags asual (the “tracking” happens).

After some weeks of surprises, we found out that a way to prevent this from happening is to have your “All Tags” Extensions (or at least those critical ones which you do not want to run twice) check for !utag.cfg['noview'] at the beginning. Because when the Extensions run for the second time (which is when the Data Layer variables relevant for the later tracking calls get set), that “noview” flag is ‘false’ (the first time when you don’t want the Extension to run, the flag is ‘true’).

Set “All Tags” Extensions to Run Before Load Rules

Another problem that arose was that we could not rely on utag.data anymore. utag.data is the object that holds the data layer after “All Tags” Extensions have run and before “DOM Ready” Extensions run. Tealium’s ABC prescribes that when using DOM Ready extensions and needing stuff from the Data Layer in it, one should refer to that data through the “utag.data” object. But with the DDLM, there is no guarantee anymore that “All Tags” Extensions really have run. So referring to utag.data["myvariable"] may return “undefined” if the “All Tags” Extension that creates “myvariable” has not run yet. In short, many “DOM Ready” Extensions all of a sudden failed to run if they relied on conditions that required variables generated in an “All Tags” Extension.

The solution was to set our dozens of extensions which ran “after load rules” (the default as per the Tealium iQ interface, not the default anymore as per Tealium support) to run “before load rules”.

The “no-arrays-and-once-per-page-only” Problem and the Need for Workarounds it Causes

Then there was and still is the most important “No-Arrays-and-once-per-page-only problem”, i.e. not being able to query more than one ID once per Pageload (even with a second fake utag.view call to simulate a second pageview, that does not seem possible) and not allowing to query the DDLM via utag.link (Tealium’s Event Tracking method).

I understand that Tealium wants to avoid people’s browsers calling their CDNs to pull data for lists with 100s and more products at the same time, but at least a couple of calls per page or some other possibility should be allowed. Because of this limit, we still have several issues:

• We write the DDLM data for each product ID the user interacts with in his session into localStorage on the first interaction so we can retrieve it later, e.g. in the cart or on purchase. This is fine because we also do not want to query the DDLM API every time the user views the same product again. But it makes cases impossible like users returning on a different device and then logging in to view a cart already filled with several products. Another case, the most severe one for us, is where the Concrete ID becomes known only after a user selects a product variant on a product page (a utag.link event without a reload of the page).
• The combination of not being able to query more products per page and having to use localStorage also means you have to deal with the general problems of localStorage, most importantly localStorage not being available in Private/Incognito mode in Safari.

All in all, the “No-Arrays-and-once-per-page-only problem” leads to relevant holes in the data we track through the DDLM API: Right now, 17% of Product Detail Views, 12% of Cart Additions, 14% of Cart Views are being tracked without product data from the DDLM API, and 3% of orders.

What Concrete Benefits Do We Get Out of DDLM?

I still believe the idea of DDLM is powerful, as it addresses real problems. Some examples:

1. We don’t have to ask developers for “yet another Data Layer variable” on the page
2. The DDLM API allows us to avoid the problem of Adobe Analytics classifications and Google Analytics Query-Time imports being ahistoric, meaning they always show the most recent value no matter what time you report on (see part 1), so you cannot use them for ephemeral data like whether a product dropped in its price, whether it is a “Christmas special” or what its current rating is. But the DDLM API can provide this temporary data, and we then just save that data to normal product dimensions (we push it all into one Merchandising eVar in Adobe and then break that one up via the Rule Builder into individual Classification dimensions to save eVars).
3. To give you a concrete example, we can now see how much impact a better price than the competitors can have on sales: Products which our price comparison engine tags with a “best price” label of “yes” (this data is, as mentioned, fed into ProductsUp) clearly outperform products where the product is worse in price (e.g. the Product Conversion Rate for Computer & Electronics products goes up by 31% (from 1.6% to 3.1%) when siroop offers the best price. We can also see that, per best-price product on the site, we earn 22.11 CHF, whereas that is only 18.23 CHF for products with higher prices than competitors (what is maybe even more interesting is that we earn 33.79 CHF per equally priced product!).
   
4. It also allows us to use such temporary product labels for retargeting, e.g. sending the information that a user has viewed a non-price-dropped product to a retargeting tool (e.g. email) so the user can be targeted with the same product again when its price dropped.
5. It also allows us to work around not being able to import data to widen product dimensions in Google Analytics, so we can use it as a workaround to report on product data that is based on the Product Concrete ID (we are using the Abstract ID as the main Product SKU, i.e. also for data imports, while the Concrete ID is only a Product Dimension which cannot be widened).

Conclusion

Leaving the integration pains in the past and looking at what we can get now, we have to admit that DDLM brings us great benefits. But due to the missing data because of the “No-Arrays-and-once-per-page-only problem”, the whole DDLM falls short of its huge potential.

I can only hope and urge Tealium (pleeeeease!!!) to improve this product before somebody else comes out with something similar, because there is truly a great idea behind it, and I will be happy to review DDLM again when it is even better suited to an E-Commerce context like ours.