Data Collection for HTML5

Throughout this document the term csaName is used to reference the designated prefix applied to all top-level scoped variables and functions within the CSA. This prefix is configurable and prevents collisions with existing variable or scripts used on the website. As such example JavaScript may include the csaName prefix where appropriate. This must be replaced with the CSA name configured for your Collection Server. In such cases the prefix appears as shown below:

The CSA can be downloaded from the Celebrus Collection Server. Ask your administrator for the Celebrus data collection URL and browse to it. The CSA is a single JavaScript file, CelebrusInsert.js which should be downloaded and hosted by the website or Content Delivery Network (CDN).

All scripts have already been minified to reduce bandwidth. Do not attempt to minify further or to edit the scripts unless directed to do so, as in doing so could corrupt them. Some methods and variables require window level scope and as such minifying these names would result in a non-working CSA.

The CelebrusInsert.js has several options you can configure using the Configuration Manager. Once configured a new CelebrusInsert.js can be downloaded from the server.

Several approaches to instrumenting the website exist, each providing their own benefits. Whichever approach is taken however the website needs to include the changes.

The script can be included within the website in one of a variety of ways depending upon how the website and its underpinning content management system is deployed. It is however crucial that the insert is included in each and every and frame of the website. Failure to do this reduces the accuracy of the data collected.

The CSA initialises automatically when the page loads, hence nothing to do.

The CSA has built in logging which can be activated at any time, by setting the appropriate logging level:

This command can be entered directly within a browser’s debug console, typically accessible by pressing the F12 key. The full range of LogLevel’s available, in order of increasing detail, are: `OFF, FATAL, ERROR, WARN, INFO, DEBUG, ALL. Setting a log level will enable all logs of that and lower levels of detail. For example a LogLevel of WARN enables WARN, ERROR and FATAL messages to be displayed.

Setting a LogLevel of DEBUG results in an additional script CelebrusLoggingUtils.js being sourced from the Celebrus Collection Server. This script provides detailed logging of event contents.

When set a LogLevel is persisted within the web browser session storage (if supported). This means that the LogLevel set is persisted through page navigations if the origin of the web page remains the same. For example books.mysite.com and music.mysite.com would have different session storage objects even though they share the same domain.

To disable logging, call:

Session storage does not survive the browser being closed so simply restarting the browser also clears any LogLevel set.

The dry run mode of operation for the CSA allows the developer to run Celebrus code in a web page without having any communication between the page and a Collection Server. With this mode of operation It is not necessary for there to be a live Collection Server running although it is necessary to make Celebrus files downloaded from a Collection Server available from a web server.

The files necessary to carry out a dry run are:

A page can be marked as being in dry run mode by including a window-level JavaScript variable set to true, like this:

where [csaName] indicates the CSA name for the Collection Server which ultimately carries out the event collection. Although it is not necessary for the Collection Server to be available during a dry run, it is necessary to download the above two JavaScript files from a Collection Server and make them available to the page from an accessible location. When CelebrusInsert.js executes and detects it is in a dry run environment it simply picks up a default set of configuration details from CelebrusLoggingUtils and carries out its normal logging output.

Again the most effective way of using dry run mode is to add the three necessary files to each test page using a tool like Greasemonkey. In this case the full script that should be included in each page looks like:

The CSA provides various API functions to allow some manual operations to be invoked by the website if that is required. A window-level Celebrus API object is defined for the page and is prefixed using the CSA Name assigned for the installation. So for instance if the CSA Name being used is "devtest" then the Celebrus API object will be

Note that in previous releases Celebrus made API functions and properties available as a series of distinct window-level references. Refer to the Celebrus API functions table to see a list of API functions and their corresponding window-level equivalents as described in previous releases.

The CSA captures any exceptions which are generated whilst it is running. If an unexpected exception occurs the CSA automatically transmits this information back as part of the data collection.

As visitors browse multiple brands of the same organisation each new domain encountered would normally result in a unique session being allocated for each. It is, however, possible to extend a session across multiple domains by using cross-domain functionality.

Enabling cross-domain functionality allows Celebrus to use its first party cookies in conjunction with third party cookies the Celebrus Collection Server sets on the collection host.

Historically default browser settings have allowed websites to set third party cookies if an appropriate P3P header has been set for the request. However the default cookie settings for Safari and Firefox currently have a more conservative approach. From a data collection point of view this change lessens the effectiveness of cross-domain. See Intelligent tracking prevention (ITP).

Google has stated as part of its Privacy Sandbox initiative, it is to drop support for 3rd party cookies within Chrome sometime in 2022. This change will prevent cross-domain functioning in its current form on Chrome (and likely Edge), meaning that events are collected within separate session contexts, one per domain a visitor traverses. Collection within multiple sub-domains of a common domain would not be impacted.

As part of this initiative Google is introducing the concept of First Party Sets, providing a mechanism for an organisation to declare the website domains it owns as belonging to the same party. Cookies marked with a new SameParty attribute are then visible between domains within the same First Party Set. As Celebrus is a first party solution, cross-domain functionality can be maintained if Celebrus is deployed as part of a First Party Set.

Support for this experimental feature can now be enabled directly within the Configuration Manager’s Deployment app.

The task of declaring the First Party Set is your responsibility. Only your organisation can determine the appropriate domains which should be included in such a set.

Further details regarding First Party Sets can be found here.

It is recommended that the Collection Server is in the same domain as the principle brand or website being monitored. For example, if you deploy a system to acme.com then collection.acme.com would be a good name for the Collection Server.

The benefits of this approach are:

All data is collected using HTTPS. Earlier versions of Celebrus allowed data collection using HTTP but this is no longer supported.

The diagram below shows a typical session. The session starts when a new browser instance requests the first page on an instrumented web page. In this example, the user may choose to navigate to other pages, as represented by the second page starting to load in the diagram. After some activity by the user there follows a period of 30 minutes of inactivity. At this point, the session expires in the Celebrus system, which causes the last page to unload and a session end is generated.

The next scenario to address is what happens after three hours of collection (the maximum session duration). In this case, the user does not navigate off page until some time later and it is only then that a new session is allocated.

Therefore the session is not ended and continues to collect events as normal. When the user next navigates to another page, the Celebrus system starts a new session. The first session ends and is summarised as normal.

These options change which attributes are present in the Web/PageLoading event (the first event from any web page). If you change any of these settings, then you must download the CelebrusInsert.js from the Collection Server and re-deploy on your website.

Use these options to capture HTTP headers seen by the Collection Servers. You can also configure the use of aggressive data collection techniques.

Variable and content rules allow specific elements of a web page to be collected. This is useful for collecting data which isn’t captured as part of a visitor’s interaction with the website (for example, data layers).

These rules can capture script variables at the window[variable name] level of a web page, and also properties of items embedded within a page. Content to be collected is identified by specifying match details for any combination of the name, id, class, tagName, href or src attributes. All matches are case sensitive. All, with the exception of tagName, support wildcards at their start and end. Alternatively, for the JS10 CSA, content to be collected can be specified using a CSS Selector, which Celebrus will use in conjunction with the standard JavaScript document.querySelectorAll(selector) function to identify which elements match.

You can configure whether variables and content are collected when first encountered, upon change, or when the visibility of content changes. Alternatively a second variable can be monitored to act as the decision point to determine when capture is to occur. Further options specify how the value is captured, using toString or JSON.stringify.

When capturing a JSON representation of a JavaScript object care should be taken to avoid reference cycles and to consider data volumes.

Meta tag attributes can be collected using the meta tag rules. You can capture the content attribute of a named meta tag, or choose to capture a specific named attribute.

Page rules are the primary mechanism for determining what data is collected from a given page within a website or mobile app. Rules are evaluated in ascending priority order. The first rule which matches determines what data is collected from that page. If no rule matches, nothing is collected.

The types of events which can be configured depend on whether the rule is targeting websites or mobile pages. A rule can have a set of exceptions associated with it. These exceptions change the configuration for specific elements on the page. For example, a rule can be configured which excludes credit card fields from collection.

For mobile apps some collection options make more sense when configured at app level rather than on a page by page basis. The options enabled on this tab are applied when the CSA starts and are maintained for the duration of the session.

Browsers provide plugins to modify or tamper with the HTTP headers as they are sent from the browser. This therefore provides the ability to simulate a visitor from another geographic location. In tandem with other techniques listed here to collect data from any website before it is instrumented, this represents a powerful approach to testing before any live changes are made to the website.

Celebrus makes use of the MaxMind geolocation databases which are embedded within the Celebrus system. Maxmind does however provide a lookup web service which can be used to determine the basic information which this database contains for a given IP address. Browse to https://www.maxmind.com/en/home.

This approach can be used to obtain data from any website which can be used as a test case. This data can be repeatedly run through the Celebrus configuration. The test case recorder within the Semantic App can be used to collect data conforming to the license of the installed Collection Server. However it also provides the capability to record from any website. The test case recorder dynamically instruments the website when it loads.

This approach can be used to test a data collection installation before the website has actually been instrumented. Browser plugins can be used to instrument a website dynamically, for any website. This only impacts the browser on which the plugin is installed and configured, requires no involvement of the website in question and is a super quick way to get some real data. This approach does need a Collection Server to be installed. Speak to your support representative if you require further assistance.

Greasemonkey is one such plugin for Mozilla Firefox which allows a web page to be modified by using small bits of JavaScript. In this specific example we’ll set-up Greasemonkey to inject the Celebrus insert into the web page when the onLoad event fires.

If the other system has a first party presence on the website, then the Celebrus session key can be readily shared. The identities of both systems are in the same scope and have visibility to one another.

If the other system has a third party to the website such as an iFrame in another domain, then this presents another hurdle, but one which is readily negotiated.

The JavaScript example below demonstrates the mechanism by creating an iFrame in the domain of the other system, placing the Celebrus session identification on the src attribute.

Because the script is created in the first domain, that of the website, access to the Celebrus session identification is available. When the iFrame executes, the browser places the cookies associated with the other system within the HTTP headers of the request. The other system’s web server then gets to see both the Celebrus identify and its own. A bridge has been built.

Celebrus measures the time taken from user navigation (click) to DOMContentLoaded if the Celebrus insert is invoked at the end of the </body> tag. If the Celebrus insert is dynamically injected when the document.onLoad event fires it measures the time taken from user navigation (click) to document.onLoad. As the previous section discusses, this is not necessarily the user experience. However trends of these measurements are extremely useful to detect unexpected changes in performance, along with relative comparisons between ISPs, geographics, browsers etc.

The Celebrus script loads and executes asynchronously to other resources so its impact is minimal, but like all web page resources, it can hold up other resources or firing of certain browser events. There is no silver bullet, it is a call on the priorities and objectives of the business.

It is possible that the inclusion of the script means that the document.onLoad doesn’t fire within the browser until Celebrus has loaded.

If the website is depending upon this event to fire before the page is made accessible to the user then consider injecting the Celebrus insert dynamically as a result of the document.onLoad event firing. By delaying the loading of the Celebrus insert, the website loading is not impacted but there is a possibility that the user interacts with the web page before the Celebrus insert is running. If Celebrus is executing personalisation then this is delayed as a consequence.

If the website document.onload event fires quickly, any delay in starting the Celebrus insert is minimal. In such cases user interactions occurring before Celebrus is able to capture them could be reasonably expected to be very low. The Celebrus script is a single file which when compressed is around 42KB. The file is static hence can be cached by the browser.

Depending on how your tool works, data collection license usage occurs when the tool executes. This is not normally an issue because the volume of test traffic compared to real traffic is small. Should this be a concern, the Celebrus system can be configured to selectively exclude this traffic from instrumentation using user agent or IP address. If such filtering is implemented, it should be noted that you are tailoring for your test tool, hence testing web page versions which are not seen by real visitors, perhaps reducing the validity of the test tool results.

From the outset Celebrus customers have always required near zero impact. To achieve this data collection agents are intelligent:

Below are two sequence diagrams describing the communication between the web server, client browser and Collection Server.

As with any browser app communicating with two or more hosts, care must be taken to ensure that failure of one of the systems does not delay page rendering. This is true for any content included in web pages, and similarly applies to Celebrus. The system includes mechanisms designed to avoid failures in the server or associated network infrastructure from impacting the performance of instrumented websites.

As defined by https://www.w3.org/TR/tracking-dnt/, a browser setting through which the user can indicate their preference not to be tracked. As this is set on the browser, it is carried with the visitor’s browsing activities across all web sites visited through its inclusion as the DNT HTTP header on all outbound requests. With the Launch of Internet Explorer 10, Microsoft set Do Not Track as its default setting. This, according to the author of the DNT standard, violated DNT’s use, because it does not protect an individual’s privacy unless the recipients believe it was set by a real human being expressing their preference.

Celebrus can be configured to adhere to the DNT browser setting, or to adhere to all browsers except Internet Explorer 10 and above, or to never adhere to the setting.

The data collection technology uses first party session cookies to maintain session context, and optionally first party persistent cookies to assist in building longer term profiles of visitors. If first party session cookies are not accepted by the visitor’s browser, the data collection silently shuts down.

The Platform for Privacy Preferences (P3P) is one mechanism which can help ensure such cookies are permitted. By using appropriate P3P HTTP response headers within the website describing the data being collected and the usage intent as per www.w3c.org/p3p visitors can make more of an informed choice.

Providing facilities to allow the visitor to opt out of data collection and advertising this within the P3P HTTP response headers can also be expected to increase cookie acceptance rates.

Integrated data collection consent mechanisms are included within the Celebrus system. There are two approaches to making use of this mechanism.

The script is composed of three distinct parts:

The script as delivered by the Celebrus servers can be included in the <HEAD> or <BODY> or each web page. The script when evaluated automatically starts the data collection.

If you wish to invoke a Celebrus API prior to it starting, for example when implementing Website integrated consent, simply place calls immediately prior to the invocation block.

The insert structure looks as shown below:

The JSON Data API provides the ability to collect data other than that which is normally collected. Such data however is out of Celebrus control, and therefore whether it contains personally identifiable information (PII) is unknown. To cater for this the JSON API has been extended to add a further API:

The privacy type is an enumeration defined at window level:

It provides the following two values:

The existing JSON API which takes a single argument defaults the DataPrivacy to MAY_CONTAIN_PERSONAL_DATA. Any data submitted through the sendJsonData API which may contain personal data is dropped unless the consent is opted in.

See the Celebrus API functions table for more details.

In circumstances where it is not appropriate to capture certain data, but a need exists to determine whether the data is the same across multiple fields (without knowing its value), Celebrus provides the ability to hash values. Hashing is lossy and should not be confused with encryption. Given a hash representing a value, is it not possible to directly reverse engineer the original value, though it is possible to guess what the value might have been and hash the guess to determine if the guess was correct.

Regulations may require the informed consent of an individual prior to data collection. The General Data Protection Regulation (GDPR) is a good example of this. Data for other use cases such as fraud detection can, in the right circumstances, be collected using legitimate interest as the legal basis.

Celebrus addresses this requirement by supporting two instances of the CSA running within the same website. One instance is for marketing and supports user consent. The other instance is for legitimate interest use cases like fraud detection and does not require user consent.

The table below demonstrates the configuration differences between the two Celebrus installations:

A text change event represents the text entered into a text component and is automatically generated by Celebrus when focus is lost from the component.

A form submit represents the state of one or more fields as they are submitted by the user.

Single page web application typically change the page URL with bookmarks to reflect their state changes. A bookmark within a URL is denoted by a #, as shown in the example below:

If the user presses the back button in the web browser, the app reverts its state to the previous bookmark. In these or similar cases, it may be required that the collected data reflect these app state changes also by generating a new page in the collected data, hence synthetically producing a navigate. To achieve this the following needs to be invoked whenever a navigate is to be synthetically generated:

Alternatively the window.location.href location can be listened to and a new page generated using the approach below:

Basket actions can be provided directly to the system as shown in the following examples.

Data provided through this API is only collected if the consent status for the session is opted in.

See the Celebrus API functions table for more details.

The CSA is highly adaptable, using discovery techniques which adapt to the browser technology. There are only a few requirements for the full data collection to operate:

Like many websites, Celebrus requires JavaScript and session cookies or session storage to operate. For those browsers with cookies, session storage, and JavaScript disabled, no information is collected by Celebrus. If this is a concern then you should consider using the HTTP API to deliver data to Celebrus. See the Data Collection for HTTP API User Guide for more information.

Web browser may decide to pre-load web pages which although haven’t as yet been requested by the visitor, are anticipated to be requested. Celebrus automatically caters for this, recognising a page is pre-loading and only commencing data collection once the page is actually requested by the visitor.

To support ITP you must deploy Celebrus so that the collection endpoints align with your website domains. For example, if you deploy Celebrus to www.acme.com then the collection host collection.acme.com is aligned. Multiple collection endpoints can be configured and deployed. In such cases Celebrus determines the most appropriate endpoint to use for each web page.

Configure collection endpoints within the collection section of the Deployment app in the Configuration Manager. The CelebrusInsert.js must be redeployed once this configuration has been updated.

Long term visitor recognition functions if any one of the following conditions are met:

All ITP based devices and Firefox prevent session continuation cross domain using cookies. However it is possible to use first party link decoration to achieve the same effect of cross domain, when directly navigating between two domains within your organisation. Link decoration places a querystring parameter on page links between your organisations websites automatically, allowing cross domain to function without cookies. Link decoration only operates on links which are listed as a Celebrus collection endpoint. The value Celebrus places as a querystring parameter is a time-limited server-side reference.

On the first page of a session, if there is no referrer that corresponds to one of the configured collection endpoints, the CSA selects the default collection endpoint, i.e. the first endpoint in the list, since this corresponds to the domain the company considers most likely to be a start point for customer sessions. Using this endoint maximises the chances for cross domain session recovery to recover a session number started on a different domain.

If the current page is not aligned to the default collection endpoint domain then ITP limits the persisted cookie expiry duration to seven days on that first page, but on navigation to further instrumented pages the system will have an opportunity to select the aligned collection endpoint and the cookie expiry duration will be set to the Visitor tracking cookie duration, as configured within the collection settings.

In summary, on the first page in a session the CSA maximises the chances for session recovery to succeed, while subsequent pages maximise cookie persistence.

A MousePath summarises the data for a series of mouse movements between a start and end point. It reports X/Y coordinates for the start and end points and uses the intermediate points captured to calculate characteristics of the path taken by the user.

Celebrus CX Vault allows personalisation to be delivered to unconsented visitors by executing a signals model on the device rather than on Celebrus servers.

See the Celebrus CX Vault User Guide documentation for further information.

It is possible to set additional HTTP request headers to be used by the Celebrus insert when communicating with collection endpoints. To do this invoke the [csaName]setHttpRequestHeader(name, value) function on the insert, to add the headers you want. This must be done for each web page. To ensure all requests are made with your additional HTTP headers in place, ensure that you set the additional headers after the script has loaded but before it is started.

The Celebrus Collection Server can be configured to include and exclude fields from data collection. This is server-side control. Another approach, or one which can be used in tandem with the server-side controls, is to express exclusions within the page delivered by the website.

Exclusions are specified by assigning one or more comma separated element IDs to the variable [csaName]collectExclude as shown in the JavaScript example below:

In the example above collection is excluded from any element on the page starting with customer.address or customer.telephone. Note that wildcards are permitted but only at either end of the identifier, and only whole segments may be wildcarded. Hence *.address.* is permitted whilst *address* is not.

Another alternate approach to exclude data collection is to add a special attribute(data-[csaname]-collect-exclude) in html input elements. This indicates the client CSA to escape events of marked elements in event stream and to exclude element data in FormSubmit events as well.

Name of the attribute depends on the csa name. If csaname is 'myTestCSA' then the attribute name is 'data-mytestcsa-collect-exclude' (all in lowercase). Here is a sample format of the html elemtent with attribute.

Apache Cordova describes itself as:

The CSA operates in a similar manner for an Apache Cordova app as it does for an HTML5 application. A few differences exist as highlighted below:

By including identifying criteria within the RSS <link> URLs, RSS feeds can be identified.

Celebrus supports multiple inserts running within the same website. This can be useful when upgrading between major Celebrus releases, as it gives you the opportunity to run with old and new Celebrus in parallel, to ease the adoption of latest Celebrus features and data. When setting up to dual run, the existing Celebrus device identifiers and user consent can be transferred from the incumbent Celebrus to the new, by using the CelebrusCopyCookies.js script. The script can be downloaded from the Collection Server, and hosted by your website. This script needs to be executed before the new Celebrus insert executes.

The JS10 Celebrus API object was introduced in version 9.7.0 and sources all of the API functions in a single JavaScript object. This was done to ensure that Celebrus functions are contained and accessed from a distinct object rather than a series of functions and variables in the window scope, and to maintain a consistent approach for naming Celebrus API functions.

As documented in previous releases, Celebrus API functions are also available as a series of window-level functions prefixed with the CSA Name assigned for the installation. These window-level functions are still fully supported by the JS10 CSA, but using the CelebrusApi object is now the recommended way to call API functions.

The following table shows current function invocations and their JS9 equivalents.

The Deployment app has options which specify whether API keys are enforced for any or all HTML CSAs, App SDK and HTTP API requests. Enforcement ensures that requests sent to the Collection Server with an invalid or missing API key are rejected.

You can use the enforcement option to introduce an API key to a website gradually. The Deployment app allows you to specify the default API Key to use in the CelebrusInsert.js file by selecting an entry from the list of the currently configured API Keys.

To roll out API keys to a website: