System Installation Operations Guide

It is usually appropriate to install the Collection Server and Real-Time Server on different physical servers. This reduces CPU and memory contention between the two types of server, reducing potential delays in responding to collection requests. Additionally, it may be desirable to isolate the collection aspect of the platform in a DMZ or similar, with minimal ports opened up between the DMZ and the internal network.

Multi-server installs follow the same basic step-by-step install procedure as detailed later. However, after the first install, ensure the servers are commissioned and running. This is the case regardless of whether or not you are installing the first of a server type in a subsequent install or not; always make sure you have successfully commissioned and started the servers you’ve installed before installing subsequent servers.

On larger sites, multiple Real-Time Servers should be installed on separate dedicated servers due to their higher memory requirements.

The JavaScript insert is included in all instrumented pages. The following recommendations ensure optimum data collection performance. The include tag should be the last item to be loaded in the web page, immediately before the body tag.

Celebrus has been tested on IPv4 stack and dual stack (IP v4 / IP v6) environments. Java prefers the use of IP v4 addresses on a dual stack environment, using IP v4 mapped IP v6 addresses. Geolocation support is provided for IP v4 and IP v6 addresses.

¹ Optional, required when collection session affinity is enabled and provided by the Collection Servers themselves. When enabled Collection Servers forward requests between themselves to ensure session affinity is maintained. This approach has the advantage of lifting the burden of session affinity away from load balancers. Some load balancers only provide load balancing using cookies, which can be problematic with some browsers and their cookie restrictions.

² Optional, required when the legacy Profile Server is in use. Real-Time Servers include Identity Graph and Profile Builder which supersede the Profile Server.

³ Optional, required when the Celebrus deployment is storing image content for personalisation. Typically your Content Management System or Content Delivery Network takes on this role.

¹ Optional, required when collection session affinity is enabled and provided by the Collection Servers themselves. When enabled Collection Servers forward requests between themselves to ensure session affinity is maintained. This approach has the advantage of lifting the burden of session affinity away from load balancers. Some load balancers only provide load balancing using cookies, which can be problematic with some browsers and their cookie restrictions.

² Optional, required when the legacy Profile Server is in use. Real-Time Servers include Identity Graph and Profile Builder which supersede the Profile Server.

³ Optional, required if you integrate your identity provider (LDAPS or OIDC) with Celebrus.

The simplest approach to session affinity is to enable Celebrus in-built support. This can be done within the Configuration Manager’s Deployment app, Collection/Web server panel. More load is placed on the Collection Servers with this enabled, but as they make use of HTTP/2 connections between themselves when forwarding requests, the cost is minimal.

Collection Servers forward requests between themselves using a trust store comprising the root certificate from the configured collection.keystore. If your keystore is self-signed and has no root certificate, all Collection Servers must use the same certificate. This is not ideal as it introduces a service break when the certificate is updated or refreshed in the future.

By default the forwarded requests between Collection Servers in addition to enforcing certificate trust, also perform hostname verification. This requires the hostname in the cluster topology to be verifiable against the collection.keystore. If this isn’t appropriate within your environment, hostname verification can be disabled within the Configuration Manager’s Deployment app, Collection/Security panel.

All Celebrus data collection technologies requiring session affinity provide an identifier in their HTTP request URI to assist with load balancing. This can be used by many ALBs to load balance traffic to your Collection Servers.

An example Nginx configuration is shown below which load balances traffic.

Key parts of this approach to highlight are the map section which extracts the load balancing identifier, and the upstream section which specifies hash $clientId consistent to provide session stickiness based on this identifier.

If you are auto-scaling your Celebrus deployment, a further concern is maintaining session affinity when scaling up and down. NginX’s map function does not maintain session affinity when the number of upstream servers changes. This concern can be addressed through third party modules such as upstream consistent hash.

Using a network load balancer is the simplest approach. It works for all uses cases with a few caveats:

Using cookies for load balancing is not a good option because it doesn’t solve the HTML5 data collection requirements for landing pages comprising multiple frames. Additionally load balancing cookies may be blocked by web browsers if the load balancer’s cookie is determined to be third party to the web page at any point. This approach should be avoided.

Celebrus can be installed on the following Windows versions:

The following Linux distributions are supported:

Celebrus servers have been tested on the following operating systems for non-production use only:

The Operations Server contains a Chromium browser in order to support testing within the Configuration Manager. As such further minimal requirements are placed upon it by Chromium for Linux platforms:

Please ensure all relevant service packs and patches are in place prior to installing Celebrus. Celebrus uses the OpenJDK Java Runtime Environment which is provided as part of the product installer.

The following guidance applies only to the profile database. Target databases supported by the Data Loader are documented separately in the Data Loader User Guide.

The Profile Server currently supports the following database versions:

The profile database requires table partitioning to be available so Oracle Standard Editions are not supported.

A keystore is used for HTTPS data collection from websites and mobile applications. For a production environment, this keystore must be signed by a known certificate authority (CA) such as https://www.thawte.com/. Such CAs are known and trusted by web browsers and mobile devices and so trust certificates signed by them.

As a general rule to reduce contentions, Celebrus recommends splitting the various storage 'areas' as much as possible amongst the physical disks. Please make sure you discuss this with your vendor. A poorly devised disk layout can seriously impact the performance of your system.

The Celebrus Platform relies on TCP/IP communication between various components within the platform (for example: between the Collection Server and the Real-Time Server. It is vital the networking infrastructure is resilient and capable of handling the volume of data that is collected.

Additionally, firewalls, load balancers, and routers must all be correctly configured to allow the various components to communicate with each other seamlessly. Failure to do so impairs the service offered by Celebrus and could impact the performance of your site (for example, personalisation). It is therefore essential that you discuss this with your vendor prior to deployment.

Prior to installation, you need to know the external IP address or domain name of the server to which Celebrus data is to be transmitted by the JavaScript insert in your web pages. This is often the address of the public facing load balancer in front of the actual Collection Servers.

Due to the increased security in Red Hat 8.7 with SELinux, Celebrus services only can be controlled (started, stopped, restarted) by a root user or a user that has sudo privileges to start the services.

If SELinux is in enforcing mode, then the following guide may help run Celebrus Servers.

Locate the relevant installer executable and execute it:

You must accept the licence terms before the install proceeds.

A secondary licence screen is displayed which covers the third-party software components utilised by Celebrus. You must signify your acceptance of these terms before progressing to the next page.

The next screen reminds you of your obligations when collecting and handling personal data (should you choose to do this), you should read and confirm your understanding of the content of this page.

Next, select the install location for the software. Typically the default location suffices. If you choose an existing product install directory, then an upgrade is performed.

Allows you to select which components to install.

Only a single instance of the Configuration Server and Profile Server may be installed within a given cluster.

On the following screen select the host name of your server.

On the cluster overview page you select your licence file and a name for your cluster. The cluster name must be at least eight characters long.

On the next page, select the cluster ports to use for Celebrus internal cluster communications.

This page configures the certificate authority keystore.

Select whether you want the Collection Server to run as a service.

Select the keystore for your Collection Server.

The keystore must be in JKS format, containing one or more entries protected by the same password. Each entry must contain a certificate and private key. You may choose to leave this information blank and set up the certificate post install.

Enables you to maintain backwards compatibility with an existing data model which requires session numbers. If you don’t have an existing Celebrus deployment, simply select No.

Enter the details for the system administrator on the Celebrus system. Your system is automatically set up with a system administrator role and user.

The system requires SMTP access in order to send password reset emails requested from the Configuration Server. Many organisations configure their SMTP servers to only send email requests from known accounts as a spam prevention mechanism. Configure the sender email address to be a known account on the SMTP server.

Select whether you want the Configuration Server to run as a service.

Select whether you want to use your own MySQL or MariaDB instance as the Celebrus configuration repository or an embedded Derby database which is managed for you.

If you select MySQL or MariaDB you’ll be asked for its details.

Select what type of profile database you require. The installer does not create the profile database, instead it creates an example SQL script in the ProfileServer/database directory. Depending on your choice of database, the SQL script is called MariaDB.sql, Oracle.sql, or MySQL.sql. This SQL script contains the statements to create the profile database. You should therefore edit and run this script after completing the install and before starting the Profile Server.

You must also configure the profile database in the Deployment app in the Configuration Manager.

Select the JDBC driver which is compatible with your Celebrus system and target database. This JDBC JAR file is copied into the correct location in the install directory by the installer.

Select whether you want the Profile Server to run as a service.

Select whether you want the Real-Time Server to run as a service:

Select the keystore for your Real-Time Server.

The keystore must be in JKS format, containing one or more entries protected by the same password. Each entry must contain a certificate and private key. You may choose to leave this information blank and set up the certificate post install.

Click Next when you are ready to install the servers.

The installer unpacks and installs the system as configured.

Once the required files are unpacked and installed, the system is configured to match the options you chose during the installation process.

If the install included the Configuration Server, the Configuration Manager User Guide found within the Documentation folder of the install should be consulted. This provides details of how to configure the system, such as users and roles. The Documentation folder also contains the release notes.

If you have a Profile Server in your deployment you must create a profile database (MariaDB, MySQL, or Oracle) if one does not already exist. A template script can be found in <install directory>/ProfileServer/database for the appropriate database type, for example MariaDB.sql, MySQL.sql, or Oracle.sql. Edit the script and run to create the profile database. If you are running this for the first time, remove any DROP statements.

Start the newly installed components, typically by using the appropriate service controls on the system. Ensure all components start correctly and view the relevant log files within the logs folder of the new install. Check to make sure no errors are reported.

Once your cluster is installed you need to commission the deployment. To do this access the Configuration Manager app via the Operations Dashboard.

Login to the Configuration Manager. Select the Deployment app from the DEVOPS section of the sidebar menu.

Add each of your installed nodes in to the deployed nodes list.

After you have added each server, click on the Commission button:

Next, select the Collection tab within the Deployment app. On the Endpoints tab configure the CSA Name and HTTPS Endpoints to match your deployment.

If you plan to use the Load Tester application to generate test traffic, click the Filtering tab and enable Allow load test traffic.

Save your changes to the repository.

The Real-Time Server upon install issues a warning indicating that no geolocation data is available. This is an in-memory database which the system uses to map IP addresses to geolocation.

To configure the geolocation database, open the Geolocation app from the Configuration Manager home page. Within the Geolocation app, browse to the Geolocation.zip file provided to you by your Celebrus vendor and save to the repository. After a few minutes the geolocation data warnings will disappear.

From the Configuration Manager home page, click on the Data Protection app.

Click the Configure button, and configure:

Press Next followed by Finish to close the wizard. Save the changes to the repository.

The table below shows the default security roles which are created when Celebrus is installed:

You should consider which users need access to Celebrus and their required permissions. The system administrator can set-up users and roles in the Users and Roles app in the Configuration Manager.

You need to configure what data is collected and from which websites and mobile applications. Collection rules are configured in the Collection Rules app in the Configuration Manager.

From the home page of the Configuration Manager, select the Collection Rules app. Select the Web and Mobile Page Rules tab. Click the Add button, and in the Add Rule wizard configure:

Click Next, the configuration page asks when this rule configuration is to be applied. Configure PageLocation as like * by performing the following steps:

The Rule summary now reads Where PageLocation like *. Press Next then Finish.

Save your changes to the repository.

For websites, you need to download the Celebrus JavaScript insert from a Collection Server.

Open a web browser and enter the URL:

You can download the JavaScript insert from this page. Further documentation for this and other aspects of Celebrus are available via the Operations Dashboard. Mobile SDKs and demonstration apps are also available here.

There are restrictions which apply to a cluster topology:

A server does not provide service until it is commissioned. The Configuration Server is always commissioned. All other servers are commissioned through the Deployment app and provide service when the Configuration Server has successfully synchronised configuration to that server.

To decommission a server, change its state to Uncommissioned. This does not take effect immediately, instead it waits for the decmomissioning time period to elapse, during which the node is said to be decommissioning.

Decommissioning is an important phase allowing the system time to migrate live traffic from the decommissioning server to others. For example, a Collection Server which sees a Real-Time Server start to decommission no longer considers that server as a target to receive new sessions.

Services can be created/removed manually by executing the relevant shell script in the install directory for the relevant server, as shown below. Due to the increased security in Red Hat 8.7 with SELinux, this needs to be performed by a root user or a user that has sudo privileges to install / remove services.

This script file uses settings in the .conf file for that server:

To determine the name of the service to be installed/uninstalled:

It is recommended that servers should be run under user accounts with as few privileges as possible to minimise the effects of the system being compromised. Ideally this user should not have a home directory and people should not be able to login as this user.

Each Celebrus cluster has a unique CA certificate and private key. This certificate and private key are collectively called the CA keystore. The CA keystore is special because it can be used to sign other certificates. This signing establishes a chain of trust which allows one server in a cluster to ensure it is talking securely to another server.

The CA certificate is only used for internal communications between servers in the cluster and never for connections from browsers or mobile applications.

You can point the installer at your own CA keystore or leave the default value. In this case, the installer creates a new CA keystore. By default, the installer creates the file called CA.keystore in the root of the install directory.

If you are installing additional servers into an existing cluster then you must use the CA keystore from the first install (which contains the Configuration Server).

If you use your own CA keystore the file must be in JKS file format. The JKS file must contain a single CA certificate, matching private key, and certificate trust chain if required. The certificates should not use aliases. The keystore and private key in the CA keystore must be protected by the same password.

The installer extracts the CA certificate into a file called cluster.truststore which it copies into the root install directory. This file does not contain the private key of the certificate authority. This file has the same password as the CA.keystore file. Do not delete this cluster.truststore file from the install directory, it is required by the servers. You also need this file when you install the Configuration Manager.

The installer updates the secure.properties file in the install directory with the cluster.truststore password:

The CA keystore is protected by a password. The installer uses this password to protect the server certificates and truststore in that install directory. In other words, all server keystores and truststore in an install directory are protected by the same password.

You can change the password on the CA keystore file when you install subsequent servers on other machines into the same cluster. This avoids having the same password protect all keystores on different machines.

The installer creates a unique server certificate for each server installation. This server certificate is signed by the cluster’s CA private key. The installer prompts you for the CA keystore when you install additional servers such as extra Collection Servers and Real-Time Servers.

The server certificate and private key is stored in a JKS format file called cluster.keystore which is located in the install directory of each server.

The server certificate is used when a server connects to other servers (for example, when the Real-Time Server connects to the Profile Server to exchange profile information). In this case the certificate is passed across so the server can identify itself. This ensures both ends of the SSL connection identify themselves with a certificate signed by the CA. The installer updates the secure.properties file in the install directory with the server keystore password:

The dashboard installer prompts for a keystore which the dashboard subsequently uses to identify itself when it connects to the Celebrus cluster and obtains status information. The main server installer creates another keystore file for this purpose, called dashboard.keystore and is located in the root directory of your Configuration Server install. You need to copy this file to the machine where you are installing the dashboard. The dashboard keystore is protected by the same password as the CA.keystore.

You require the CA truststore file when you install the Configuration Manager. This is required because the Configuration Manager must trust the server certificate provided by the Configuration Server.

The installer prompts for a keystore which you provide for your Collection Server. This step is optional and you may choose to configure this post install. This keystore is used by external applications which send data to your Collection Servers (typically websites and mobile applications). If you configure a keystore file, it is copied into your Collection Server directory and renamed to collection.keystore.

The file must be in JKS format and contain the certificate, matching private key, and certificate chain. The keystores and private keys in the keystore must be protected by the same password. The installer updates the secure.properties file in the install directory with the Collection Server keystore password:

You can update the collection.keystore file with new certificates and the file is automatically reloaded within a minute. This enables you to change server certificates without restarting the Collection Servers.

The installer prompts for a keystore which you provide for your Real-Time Servers. This step is optional and you may choose to configure this post install. This keystore is used for external connections to the Real-Time Server from browsers and mobile apps when requesting content for personalisation from the Real-Time Rules content library. If you configure a keystore file then it is copied into your Real-Time Server directory and renamed to realtime.keystore.

The file must be in JKS format and contain the certificate, matching private key, and certificate chain. The keystore and private keys in the keystore must be protected by the same password.

The installer updates the secure.properties file in the install directory with the Real-Time Server keystore password:

You can update the realtime.keystore file with new certificates and the file is automatically reloaded within a minute. This enables you to change server certificates without restarting the Real-Time Servers.

The following properties are set within the secure.properties file. This file is located in the install directory and contains common properties which apply to all applications and servers under that directory.

MySQL keystore / truststore name defaults to mysql.keystore and mysql.truststore and by default the system looks in the server install directory then in the root install directory before trying the value of the use.ssl.db.mysql.keystore for valid path information.

The five MySQL SSL properties have equivalent names for MariaDB (for example, use.ssl.db.mariadb and use.ssl.db.mariadb.keystore).

The use.ssl.db.oracle property applies to the Real-Time Servers and Profile Server only. The use.ssl.db.mysql property applies to both Real-Time Servers, Profile Server and Configuration Server.

The available and enabled cipher suites are output to a server’s log file on start-up, when SSL is enabled. The default set of ciphers is subject to change between software releases as recommendations change.

Override by setting the javax.net.ssl.cipherSuites system property with the server’s .conf file. This cannot be set within the secure.properties file because it is a built-in Java Virtual Machine parameter.

The available and enabled protocols are output to a server’s log file on start-up when SSL is enabled. The default list of enabled protocols is TLSv1, TLSv1.1, and TLSv1.2.

Override by setting the javax.net.ssl.protocols system property with the server’s .conf file. This cannot be set within the secure.properties file because it is a built-in Java Virtual Machine parameter.

To configure MySQL to support SSL communications.

A default install of Oracle 11g is configured with advanced security. This includes the required orapki command line utility which is required to create the self-signed certificate and key.

Below is and example list of all the installer properties:

The table below outlines the properties in more detail. The Used by Servers column indicates which properties are required when installing which servers. If using the --disable-components command line flag to install only selected servers, you need only include properties for those servers or those marked as All. Note that there is no harm in providing properties for other servers in the file, they will just remain unused (which makes it easy to mostly re-use the same properties file to install several different servers).

Begin by stopping the Configuration Server and taking a backup of the derby directory in the Configuration Server install directory. This is very important should you need to rollback an upgrade because of an unexpected problem. Prior to Celebrus 9.5, Rolling upgrades are not supported. Downtime of the Celebrus cluster is required to perform the upgrade. Choosing a quiet part of the day is beneficial to reduce service disruption. The installer stops all services during the install process. If your servers are running as services then the installer re-registers them during the install process. This is required because the service name often includes the product version number which changes between releases.

Rolling back an update is a manual process. You must uninstall the product and re-install the earlier version. It is for this reason Celebrus recommends you take a backup of the derby directory in your Configuration Server before starting. After you have re-installed the earlier version, restore this derby directory then you can start the Configuration Server.

Every Celebrus install has a Java Virtual Machine (JVM) included in it. The JVM provided with the product install is located in the jre directory. Java moves forward with regular releases as does the version of Java provided with a Celebrus install. Therefore, when you upgrade a Celebrus install you may also get an upgraded Java Virtual Machine.

Introduced as part of Celebrus 9.5, rolling upgrades permit an existing Celebrus deployment to be upgraded one server at a time, without the need to stop the entire cluster. This hugely reduces downtime.

Important points to note:

To perform a rolling upgrade, all Real-Time Servers must be upgraded before any Collection Servers. Other servers can be upgraded in any order.

The recommended order is:

It’s important to ensure that once an upgrade is complete everything is working correctly. Some suggested checks are listed below:

The Deployment Manager is tool which, among other things, can automate upgrades to your Celebrus deployment by executing an installer for each cluster install as well as the Operations Server and dashboard. The tool supports rolling upgrades if the Real-Time Servers were not co-installed with either the Collection or Profile Server. The options selected within the tool can be saved to support execution in headless mode.

Using an S3 Event Store requires the following permissions. As the names of these permissions may change as the Amazon API and SDK changes, they are listed in abstract terms.

You will also need whatever other permissions are required based on how the bucket is being accessed. E.g. the GenerateDataKey permission.

The Event Store configuration is stored in the system repository. Changes can be viewed and tracked in the Audit Trail app. Users must have write permission to the Event Store before they can change configuration.

The Real-Time Server creates a default events directory under it’s install directory. This directory location can be changed in the Event Store app and applies to all Real-Time Servers. For example, if the directory is set to /events then every Real-Time Server uses a directory called /events to store the files. If this events directory is left blank in the Event Store app then the default events directory is used.

If you have old event stores from a decommissioned server and wish to be able to query them (e.g. to create test cases and event samples) or reprocess over them, then you can add these within the Event Store configuration. Nothing will be written to them, but they will be available for reading. Note that the directies you specify can be anywhere, but they must be in the correct format; that is a directory with an export folder inside which contains the Event Store files.

Additional diagnostic information is available for all Celebrus application components within their appropriate log files located within the <install directory>/logs directory. Server log files are limited by size. Once a log file reaches a threshold size, it is renamed and appends a number to the name then creates a new log file using the original name. By default, ten such log file iterations are retained before the oldest is overwritten.

The installation of Metabase is designed to be as simple as possible and can typically be performed in a matter of minutes.

This section details simple steps that should be performed after an install to verify the application components are operating correctly.

This section explains how to perform headless installs of the Metabase software.

The Metabase installer executable is built using BitRock InstallBuilder. The installer can be run as a command line application with a -optionfile <full path to options file> option. In this mode the installer displays no user interface and configures the product install entirely from the contents of an options file.

The following shows an example command which launches a headless install on a Windows server. Note that options on both Windows and Linux use --. The example also uses the Windows line continuation character ^ so the command can be run as shown from the command line or from a batch file.

The return code from the installer process indicates success or failure. A zero return code means the install succeeded. Anything else indicates the install failed.

The --prefix command line option selects the install directory.

When a UI installer runs, it writes its install options (plus some extra values) out to a file installer.properties. This file is found in the metabase folder in the product install directory. This is very useful to determine the correct value of an install option.

The installer options file can be used across all the supported platforms (Windows and Linux). However, the paths must be appropriate to that platform. This particularly affects the jettyKeystoreFile and dashboardInstallDir properties and the --prefix command line option.

The Metabase Server installer performs upgrades. You run this installer in the existing Metabase install directory. You can also run the installer process as a headless application (unattended mode).

The upgrade does not modify the metabase.properties configuration file. The metabase server automatically upgrades its database if necessary on the server start-up.