Building a Custom Servlet for SDL WorldServer

SDL WorldServer allows customers to expand WorldServer capabilities through pluggable components. There are several exit points in WorldServer through which custom code can be invoked to perform customer-specific operations.

Here are the types of pluggable components currently supported:

• AIS Triggers: AIS stands for Asset Interface System and is the framework WorldServer uses to access content whether that content is in a content management system, a file system, a database, etc. Through a customized AIS trigger, it is possible to execute customer-specific actions in response to specific AIS-related events.
• Automatic Actions: customized automatic actions to be executed within a WorldServer workflow.
• Authentication: can be used to override the default password validation strategy.
• Filters: allow the support of new types of assets within WorldServer.
• Notification Service: allow email messages to be customized.
• Servlets: customized WorldServer-aware servlets.
• TM Services: TM stands for Translation Memory. It is possible to provide alternative services to be used in place of the exposed TM services.

Developers use the SDL WorldServer Software Development Kit (SDK) to write these pluggable components. The implementation begins by extending an abstract class corresponding to the type of component you want to create. For example, the com.idiominc.wssdk.component.servlet.WSHttpServlet abstract class needs to be extended in order to implement a customized servlet.

The abstract classes provide us with a handle to the WSContext object, which provides us with access to the service managers.

I've implemented a very simple servlet for demonstration purposes. I'm using Maven in my project, so I first installed the wssdk-server jar file in my local repository using the following.

mvn install:install-file -Dfile=wssdk-server.jar -DgroupId=com.idiominc -DartifactId=wssdk-server -Dversion=11.1.0.3909 -Dpackaging=jar

There are only two dependencies in the pom.xml file.

<dependencies>
    <dependency>
        <groupId>com.idiominc</groupId>
        <artifactId>wssdk-server</artifactId>
        <version>11.1.0.3909</version>
    </dependency>
    <dependency>
        <groupId>javax.servlet</groupId>
        <artifactId>javax.servlet-api</artifactId>
        <version>3.1.0</version>
        <scope>provided</scope>
    </dependency>
</dependencies>

The ProjectInfo servlet outputs project information from all projects available in the system (active and inactive). It demonstrates the use of the user and workflow managers for gathering information.

Deploying the custom servlet

The steps for installing the custom servlet include:

1. Creating a WorldServer component deployment descriptor
2. Creating a component archive (zip file) containing the component classes and, if required, including all of the non-WorldServer library dependencies.
3. Using the WorldServer UI to deploy the component.

The component descriptor is an XML file that provides WorldServer with the information needed to identify and register the uploaded component. Only one descriptor file, named as desc.xml, is provided per deployment. If your project contains multiple implementations (i.e. servlets, filters), all references are included within a unique descriptor file.

The contents of my project's desc.xml is shown below.

<components>
    <servlet class="com.worldservertraining.ProjectInfo"/>
</components>

Create a zip file containing the XML descriptor file and your classes. Note that the desc.xml file must be at the base of the zip file. In addition, your Java classes must be placed in the zip file under the appropriate Java package structure. There cannot be any jar or zip files inside your deployment zip file. In order to use supporting libraries in your projects, you can either expand these supporting libraries in your zip package or have them added to WorldServer's classpath.

Once the zip package is ready, you can use WorldServer's UI to deploy your customizations. Your WorldServer user will need to have the right privilege to perform the following steps.

Login to WorldServer and click the Management option.

Click the Administration option.

Click the Customization option.

Select the appropriate component type and click the Add button to upload your zip package. If your deployment includes components of different types, select any option from the drop-down. WorldServer will workout the components from the descriptor file anyway.

Once uploaded, providing no errors are displayed, you should see your customizations listed as shown below.

The Name, Description and Version properties are taken from the respective properties in the bespoke component object.

Testing the custom servlet

Of course testing depends on the type of component you have deployed. For instance, an automatic action (auto_action) component will require an association with a workflow step for testing to work.

Invoking a servlet involves requesting the following URL.

http://localhost:8080/ws-legacy/ws_ext?servlet=<servlet_name>

Therefore, our ProjectInfo servlet would be requested using:

http://localhost:8080/ws-legacy/ws_ext?servlet=project_info

SDL Web Deployer Extension for sending Purge requests to Akamai

Akamai provides a simple REST API for automating content purge requests. For detailed information regarding the API, please refer to the CCU REST API Developers Guide.

Akamai's REST API supports a number of calls. In this article I will only cover the calls to:

• Make a Purge Request
• Check a Purge Status

I've implemented an AkamaiRestFlusher in the form of a Deployer Extension. When a page in SDL Web (Tridion) is sent for publishing, this extension triggers the Akamai purge automatically.

A properties file named publications-domain.properties provides the list of public website base domains known by Akamai keyed by publication id. For example:

91=http://www.firstexample.com
93=http://www.secondexample.com
96=http://www.thirdexample.com

The deployer extension takes the data available from the deployment package, like the publication id and page URL Path, to construct the page URL known by Akamai.

In addition to constructing the page URL for purging, the extension also constructs a list of binary URLs referenced by the page. The purge request will include the page URL and all binary URLs in one call.

The AwaitPurgeCompletion property of the deployer extension allows the administrator to determine whether the publishing task should hold and wait until the entire purge completes before continuing. In case this is set to true, the extension utilizes the pingAfterSeconds attribute, available in the response from the purge call, to halt the processing before submitting a call to check the purge status. The extension continues submitting calls to check the purge status until the purge completion is confirmed or the maximum waiting time of 30 minutes is reached.

Akamai purges usually take time to complete, so the general advise is to set the AwaitPurgeCompletion property to false.

All communication is performed using JSON.

SDL Web Experience Optimization / Fredhopper Implementation Pitfalls

In this post I would like to address some issues experienced during a recent implementation of the SDL Web Experience Optimization. Experience Optimization integrates SDL Web with Fredhopper and supports creating and managing targeted content for websites driven by SDL Web. For more information, please refer to the online documentation.

Let's set the scenario:

• 200-300 market websites delivered from SDL Web
• 400 promotions already created (at the time of the experienced issues)
• 6 live attributes set in Fredhopper
• A plan for 1000 promotions in the coming 12 months

Fredhopper response size

Fredhopper stores all its configuration in XML files. In our case, there are hundreds of promotions using the BluePrinting capability with “Include child publications” selected. This results in one XML node per promotion containing a vast list of publication ids. All these get serialized to a single XML leading to a file of 25 MB.

With compression enabled, the 25 MB response gets shrunken very well (to ca. 800 K), however parsing of an XML document of such size is an expensive operation. This has an impact on the webservice which parses the XML from Fredhopper before returning to the SDL Web Experience Optimization (SmartTarget) UI.

A hotfix (ST_2014.1.0.2236.zip) was provided by SDL R&D which brought a considerable improvement in the listing of promotions in the UI. The completion time for the GetPromotions call came down from 48 seconds to 15 seconds.

Automatic promotion publishing

Automatic publishing is enabled by default. Every time a promotion is created or updated in SmartTarget, the updated business.xml file (together with a few other files) are made available inside a directory so the syncserver/syncclient processes can push them to the query instances. In addition, the same files are copied to another directory, so they appear in the history.

In other words, not only the new promotion is saved in Fredhopper but also becomes automatically available to the query instances, and therefore the front-end client applications (i.e. websites).

Turning off automatic publishing avoids these intensive publishing operations at save time also saving on server resources. At save time, the business.xml file is still immediately saved on the indexer instance, but there is no longer any file copying for the syncserver/syncclient processes.

Disabling automatic publishing in Fredhopper requires adding an <ApproveEveryChange>false</ApproveEveryChange> node under the <IndexServer> node in the smarttarget_conf.xml file

Nothing changes in the use of the SmartTarget UI. Newly created and updated promotions are shown instantly. The only difference is that these promotions do not become automatically available to the front-end apps. They would appear as “pending” inside the Fredhopper admin UI.

The change described above requires an additional promotion publishing process. There are two options regarding this:

1. An administrator publishes the pending promotions from the Fredhopper Business Manager.
2. Publishing via a scheduled job which uses the com.tridion.smarttarget.webservice.ApproveAllChangesWorkerThread class (available from the smarttarget_core.jar) to perform the work.

With automatic publishing disabled, the time taken to save a promotion in SmartTarget came down from 1.2 min to 10 seconds.

Reload interval on query instances

The previous section relates to the synchronization of published configuration (XML files) from an indexer instance to the query instances, which is done by the syncserver/syncclient processes.

It is also possible to configure the interval in which a query instance will reload these configurations. The reload is done by the qserver process and involves reading the synchronized files and updating the rule engine and in-memory data structures. This qserver process also serves the end-user requests, so it's important to set the reload interval to a suitable time to avoid overload.

The update frequency can be updated via a property inside the query instance's config/system.xml file. The default value is one minute only.

Add the following into the system.xml (under the "root" node):

<node name="com"><map/>

  <node name="fredhopper"><map/>

    <node name="util"><map/>

      <node name="prefs"><map/>

        <node name="BusinessProperties">

          <map>

            <entry key="auto-check-for-updated-config-interval" value="600000" />

          </map>

        </node>

      </node>

    </node>

  </node>

</node>

The configuration above puts the update interval to 10 minutes.

Notes:

• After changing the configuration it is better to restart the qserver process.
• Make sure to configure the same on all the query instances.

After this change, no matter how frequent the configuration (XML) gets published, a query qserver will only look for it every 10 minutes.

The change can be verified in the qserver.log file by checking the time interval in which the "Finished reloading business settings" message appears.

Promotion BluePrinting

This section is not meant to provide a recommendation, but only to share our experience in this particular project. From time to time you may find the need to re-index all content published from SDL Web (Tridion) to Fredhopper. In fact, according to Fredhopper's team, a full re-index is something that should happen fairly frequently.

Now imagine having 200 to 300 websites to re-publish from SDL Web in order to trigger the re-indexing against a wiped Fredhopper indexer. This is just not feasible.

As explained in the SmartTarget online documentation, it is possible to create generic promotions applicable for all Websites at a higher level in the BluePrint so that website publications lower down can use them.

So you can avoid indexing content from 200 to 300 lower level publications and having all content indexed from one publication. This means that re-indexing would require republishing content from a unique higher level publication in SDL Web.

There are two ways to achieve this:

1. Localizing the component templates at the lower level publications and removing the "Add to SmartTarget" TBB.
2. Adding conditional logic to the TBB in order to only process content originating from the higher level publication.

Of course this solution imposes limitations (i.e. if a promotion should only ever apply to a specific market), but it was a good fit in this particular project.

Tuning

After a trial and error period, the following memory settings were implemented to meet the ongoing server load:

• 64 GB of RAM on Fredhopper servers running an indexer and query instances.
• 32 GB of RAM on Fredhopper servers running a query instance only.

Wiping Fredhopper indices

You can find the single instruction for wiping/deleting indices in the online learning center (https://www.fredhopper.com/learningcenter), but this post aims to detail all the steps involved in a real world scenario.

Consider a fredhopper installation with two instances: indexer and query.

1. Ensure that both instances are stopped

2. Execute the following for deleting the indices

sudo su - fredhopper
cd /home/fredhopper/fredhopper

./bin/deployment-agent-client --location <HOST> delete --instance indexer data/indices

./bin/deployment-agent-client --location <HOST> delete --instance query data/indices

3. Backup the processed XML files

It is very likely that the /processed/batch/ directory contains hundreds of files, which could result in the "Argument list too long" error when executing the cp or rm commands. An alternative approach using "find" is presented below.

mkdir /tmp/fredhopper-xml-backup/batch
cd /tmp/fredhopper-xml-backup/batch

find /home/fredhopper/fredhopper/data/instances/indexer/data/xml/processed/batch/ -maxdepth 1 -name "*.xml" -exec cp -uf "{}" . \;

4. Remove the processed XML files (IMPORTANT)

find /home/fredhopper/fredhopper/data/instances/indexer/data/xml/processed/batch/ -maxdepth 1 -name "*.xml" -exec rm -f "{}" \;

5. Restart the instances (indexer and query)

cd /home/fredhopper/fredhopper
./bin/instance indexer start &

Now publish content from SDL Web (Tridion) to be indexed in Fredhopper. Check http://<HOST>:<INDEXER_PORT>/fredhopper/sysadmin/indexinfo.jsp to confirm the published content has been indexed.

./bin/instance query start &

Check http://<HOST>:<QUERY_PORT>/fredhopper/sysadmin/indexinfo.jsp to confirm the indexed item has reached the query instance.

Building a distributed SDL Web Experience Optimization / Fredhopper solution

This post describes a solution for creating a distributed Fredhopper setup for a real-world Production environment. This solution is composed of the following Fredhopper instances:

PRODUCTION "PROD-ORIGIN" INSTANCES

• Fredhopper ProdOrigin Indexer (on "prod-origin-01" server)
• Fredhopper ProdOrigin Query 1 (on "prod-origin-01" server)
• Fredhopper ProdOrigin Query 2 (on "prod-origin-02" server)

PRODUCTION "PROD-REPLICATION" INSTANCES

• Fredhopper ProdReplication Indexer (on "prod-replication-01" server)
• Fredhopper ProdReplication Query 1 (on "prod-replication-01" server)
• Fredhopper ProdReplication Query 2 (on "prod-replication-02" server)

Consider all these instances running on Linux servers with the ProdOrigin and ProdReplication instances available from separate data centers.

There is no "out of the box" mechanism for replicating the published configurations (XML files) between the two Production environments. A bespoke automated script is registered on both Production Indexer instances as follows:

cd /home/fredhopper/fredhopper
./bin/deployment-agent-client set-option indexer /com/fredhopper/config/VersionControl/@customer-publish-action=/fredhopper/indexer/bin/post-publish.sh

This script copies the published business.xml to the remote instance and reloads it by invoking the http://${TARGET}:${HTTP_PORT}/fredhopper/sysadmin/reload-config.jsp?select=business URL.

In order for the script to work, a passwordless authentication must be established between the ProdOrigin and ProdReplication servers.

Parameters such as "target", "user" and "httpport" are set in the post-publish.sh script, which then uses these parameter values to invoke the replicate-smarttarget-promotions.sh script. Both these scripts are available from:

post-publish script
replicate-smarttarget-promotions script

Validation

1. In the SDL Web/Tridion CME, navigate to Targeting

2. In the top left, the source drop-down should default to Staging

3. Validate that both the Staging and Production sources load their lists of promotions correctly

4. On source Staging, create a new Promotion called "TESTPROMO" and save it

5. Click the ... button of TESTPROMO and choose Copy To... > Production

6. Change the source drop-down to Production, and validate that TESTPROMO is now available in Production

7. Log in to the ProdReplication Fredhopper server, and validate that the TESTPROMO has been copied by the replication script:

ssh root@prod-replication-01

grep "TESTPROMO" /fredhopper/data/instances/indexer/config/business.xml

If the promotion was replicated, you'll see an XML statement on the command line.

Conclusion

The standard Fredhopper processes continue working as usual on the Production servers. For example, the syncserver/syncclient processes continue sending the published configuration (XML files) from the indexer to the query instances. The replication itself is taken care of by the bespoke replication script as soon as the origin indexer instance is updated. As expected (and required), the replication is between indexer instances and Fredhopper takes care of the internal processing as usual.

Exploring the SDL Mobile Devices Database

It is possible to query the SDL Mobile Devices database by using a tool such as RazorSQL.

• Download and start RazorSQL
• Select Connections/Connect from the main menu
• Click the Add Connection Profile tab
• Select HSQLDB / HyperSQL as the database type as shown below.

• Click Continue
• Set a Profile Name of your choice
• Enter value "sa" as the Login
• Leave the Password empty
• Click the browse button to select the Database Script File
• Select the sdl-context-repository script file as shown below.

Your new connection profile details should now appear as follows.

Click the Connect button to access the database. You can start querying the devices database. Below are a few examples.

Select the device names per vendor showing the default width/height information

Select the list of OS names

Select the list of Browser names

Conclusion

The ability to query the devices database helps to find out which values will be set in the ADF content store. This may assist the creation of conditional logic in your implementation as well as setting up Promotions in SDL Experience Optimization (SmartTarget).

Connecting to SDL Media Manager

The solution presented here assumes a Secure Token Service (STS) is readily available. The helper class below can be used for connecting to SDL Media Manager with a secure token.

Here is an extract from the App.config file containing the bindings and endpoint information.

Note that TENANTNAME is the name of your company used by Media Manager.

Uploading images from SDL Web (Tridion) to SDL Media Manager

Consider the scenario where you need to migrate all the images from your SDL Web CMS to SDL Media Manager. For each image in SDL Web, the solution would involve:

• Downloading the image
• Uploading the image to SDL Media Manager
• Creating a program and distribution in SDL Media Manager
• Putting the distribution Live
• Finding the "using components" in SDL Web and updating each one by replacing the relationship to the image in SDL Web with the relationship to the equivalent image in SDL Media Manager.

To complete all the steps above, you will require the following.

• SDL Web Core Service client
• SDL Web Stream Download client
• SDL Media Manager client
• ECL Service client

Here are the supporting app settings.

As shown above, the first set of properties is related to the SDL Web (Tridion) Core Service. The second set of properties is related to Media Manager, which includes the IssuerName specifying the URL to the Secure Token Service (STS). The third set of properties is required for specifying:

• A spreadsheet mapping the SDL Web (Tridion) folder structures to the equivalent folder structures in Media Manager.
• A directory used for storing the images downloaded from SDL Web for uploading to Media Manager.
• The tcmid of the schema (asset type) to be migrated. In the listing above, the tcm:5-33-8 corresponds to the Image multimedia schema.
• The name of the asset type (as defined in Media Manager) to be migrated.
• The named of the outlet (as defined in Media Manager) to be used when creating the distributions in Media Manager.
• The name of the tag (as defined in Media Manager) to be used when uploading and creating the assets in Media Manager.

The Media Manager Secure Token Service is essential for running the asset migration. It allows the communication to Media Manager to be established through the API. In the application settings above, the STS service is available from the http://localhost:89/IWSTrust13/ URL.

The Media Manager API does not support the creation of folders. Therefore, before migration can take place, the folder structures in SDL Web must be replicated in Media Manager to accommodate the assets. The TridionToMediaManagerFolderMappingsFile configuration property specifies a spreadsheet containing the full mappings between the existing folder structures in SDL Web and the equivalent folders created in SDL Media Manager.

Downloading assets from SDL Web (Tridion)

The code for downloading assets from SDL Web is shown below.

Uploading to SDL Media Manager

The code for uploading to SDL Media Manager is shown below. This code also creates the program, the distribution and puts the distribution live.

Updating the relationships in SDL Web (Tridion)

The download and upload steps complete the process of migrating assets from SDL Web to SDL Media Manager. However, it is also important to update the components using the migrated asset by swapping the reference for the new asset in SDL Media Manager.

This step relies on a SessionAwareEclServiceClient only available through a netTcpBinding which is not exposed. Therefore, a solution invoking this service can only run on the CME server itself.

And through the following endpoint.

You will need to copy the Tridion.ExternalContentLibrary.Service.Client.dll file from the SDL Web CME server and import into your solution in order to build the project.

The code below takes the core service and ECL clients, the migrated SDL Web multimedia component and a Dictionary object containing information gathered during the upload process. This information includes the distribution id, which is used to create or get the stub to the asset originating from Media Manager.

Adding a list of SDL Web (Tridion) components to Workflow using the SDL Web Core Service in Java

Here is a code snippet demonstrating how a list of updated components can be added to a new Workflow instance in SDL Web using the Core Service API.

First step to building a Java SDL Web (Tridion) Core Service Client

I've recently written a SDL Web (Tridion) Core Service client in Java. I was able to get started quite quickly by using the jaxws-maven-plugin as shown below.

The classes will be generated inside the /target/generated-sources/wsimport/ folder in your project. You only need to assign this folder as a source folder in your IDE and start coding!