Endeca Logging And Reporting System

Here We will explore the endeca logging and reporting system. Reports are great tool to capture customer behavior on application usage.

Like :

• Most popular search on application ?
• Search conversion rate ?
• How users are browsing the site ?
• Search and navigation traffic on site ? 
   

Steps to add logging feature in application.

1. Add Logging requests in application code.
2. Modify Report Settings file if required.
3. Configure Log Server and Report Generator.

How it works ?

1. Send Logging Requests to LogServer using Logging API.
2. LogServer write logging requests to log files.
3. Report Generator reads log files and generate reports.

Key Components of logging and Reporting System.

1. Logging API : Used to Create and submit log requests. Below are two core classes of Logging API.
    a. LogEntry : Key/value pair container for log requests.
    b. LogConnection : Used to connect with LogServer.

Log requests are independent of MDEX Engine requests.

2. Log Server : Translate application log requests to flat file.

3. Report Generator : Generates Endeca reports using below three files.

     a. Log File (generated by log server).
     b. Report Settings File. Default Path , %ENDECA_CONF%\etc\ report_settings.xml.
      c. XSLT Stylesheet. -- Setting to specify whether to generate XML or HTML reports. Default Path , %ENDECA_CONF%\etc\tools_report_stylesheet.xsl

Note : In the case settings file path for a report generator is specified in workbench that settings will be effective.

Customizing Report : You can customize reports.  Below are the most common customization.

1. Disable report sections or items.
2. Setting default and list for Top N items drop down.

Here is the report with all the sections and items.

Report after disabling Sorting section and Top Search Key item.

Here is the report with default Top N value (20).

Report after customizing it to 25.

Report Setting file used for above customization.

=======================================================================
<report_settings>
    <top_n report_item="top_search_terms" n="5, 10, 25, 50" initial_value="25" />
    <disable list="sort, top_search_keys" />
</report_settings>
=======================================================================

Endeca Implementing a Custom Java Record Manipulator

Follow below steps to write and configure custom java record manipulator.

1.   Write java manipulator using CADK (Content Adapter Development Kit) API classes.

2.   Compile this java manipulator into a JAR file.

3.   Configure manipulator in Forge pipeline.

Below is the example of simple java manipulator.

This record manipulator is doing two tasks.

1. Converting/updating one record property to lower case.

2. Adding one new record property.

=========================================================================

 package com.test.endeca.manipulator;

import com.endeca.edf.adapter.Adapter;
import com.endeca.edf.adapter.AdapterConfig;
import com.endeca.edf.adapter.AdapterException;
import com.endeca.edf.adapter.AdapterHandler;
import com.endeca.edf.adapter.PVal;
import com.endeca.edf.adapter.Record;

public class SimpleJavaManipulator implements Adapter {

    private static final String PASSTHROUGH_SOURCEPROP = "SOURCEPROP";
    private static final String PASSTHROUGH_TARGETPROP = "TARGETPROP";
  

    @Override
    public void execute(AdapterConfig adapterConfig, AdapterHandler adapterHandler)
            throws AdapterException {       
        String sourceprop = adapterConfig.first(PASSTHROUGH_SOURCEPROP);
        String targetprop = adapterConfig.first(PASSTHROUGH_TARGETPROP);
      
        if (sourceprop == null)
            throw new AdapterException(PASSTHROUGH_SOURCEPROP + " passthrough not specified");

        if (targetprop == null)
            throw new AdapterException(PASSTHROUGH_TARGETPROP + " passthrough not specified");

       
        for (int inp = 0; inp != adapterHandler.getNumInputs(); inp++)
        {          
            boolean hasMoreRecords = true;
            long currentRecord = 0;
          
            while(hasMoreRecords)
            {
       
               Record rec = adapterHandler.getRecord(inp);
               if (rec != null)
                {
                    ++currentRecord;
                    String sourcepropValue = null;

                   
                      for (PVal prop : rec)
                      {
                          if (prop.getName().equals(sourceprop))
                          {
                               sourcepropValue = prop.getValue();

                              // update current property to lower case
                              prop.setValue(sourcepropValue.toLowerCase());
                              break;
                          }
                       }
                    // Add new property
                    if (sourcepropValue != null)  {
                        rec.add(new PVal(targetprop,sourcepropValue));                                               
                    }                  
                    adapterHandler.emit(rec);
                } else {
                    hasMoreRecords = false;
                }
            }
        }      
    }

}

=========================================================================

Note : CADK API jar is available at %ENDECA_ROOT%\cadk\lib (adapter.jar).

Once manipulator is implemented and compiled into jar, now configure this manipulator into forge pipeline.

Steps to Configure java manipulator into forge pipeline.

1. Open Endeca application pipeline in developer studio.

2. Right Click then select New then Java Manipulator.

3. In manipulator pop up add general information like java home, manipulator class name and path of the manipulator compiled jar.

4. Add data sources.

5. Add Pass Throughs.

6. Add manipulator as data source in next pipeline component.

7. Now your manipulator is ready. Save the pipeline and run the baseline.

Endeca SEVERE: Unauthorized (401): Unauthorized access to workbench

Endeca application initialization failed with below error.

=======================================================================
[01.23.16 10:12:03] SEVERE: Unauthorized (401): Unauthorized access to workbench. Please check your credentials.
Occurred while executing line 3 of valid BeanShell script:
[[

1|
2|
3|   IFCR.provisionSite();
4|   CAS.importDimensionValueIdMappings("MyStore-dimension-value-id-manager",
5|                   InitialSetup.getWorkingDir() + "/test_data/initial_dval_id_mappings.csv");
6|

]]

[01.23.16 10:12:03] SEVERE: Caught an exception while invoking method 'run' on object 'InitialSetup'. Releasing locks.

Caused by java.lang.reflect.InvocationTargetException
sun.reflect.NativeMethodAccessorImpl invoke0 - null
Caused by com.endeca.soleng.eac.toolkit.exception.AppControlException
com.endeca.soleng.eac.toolkit.script.Script runBeanShellScript - Error executing valid BeanShell script.
Caused by com.endeca.soleng.eac.toolkit.utility.IFCRUtility$HttpStatusException
com.endeca.soleng.eac.toolkit.utility.IFCRUtility execute - Unauthorized (401): Unauthorized access to workbench. Please check your credentials.
 
=======================================================================

Root Cause : This happened due to ToolsAndFramework installed with password [admin@1234] other then admin.

Update ToolsAndFramework password to admin using below steps.

C:\Endeca\ToolsAndFrameworks\11.1.0\credential_store\bin> manage_credentials.bat add --key ifcr --user admin

Enter password for user admin :

Re-enter password to confirm :

Credential of type [password] already exists for this key.

Do you want to replace it [yes/no] ?yes
23 Jan 2016 10:27:23,954  INFO CSFHandler:146 - Credential successfully updated for map : endecaToolsAndFrameworks.

C:\Endeca\ToolsAndFrameworks\11.1.0\credential_store\bin>

ProductCatalogSimpleIndexingAdmin component indexing phases and indexing buttons are missing

Sometimes when you are working on indexing related components. You may face this issue. In this case ProductCatalogSimpleIndexingAdmin component UI is looks like normal ATG component [indexing phases and indexing buttons are missing].

Below is the screen capture of erroneous ProductCatalogSimpleIndexingAdmin UI.

Root Cause : This happens when any one of the component listed in phaseToPrioritiesAndTasks is not able initialize/start.

Solution : Make sure that there is no error in all components in phaseToPrioritiesAndTasks .

After fixing issue here is the expected UI.

Write to record store failed due to handle invalid record

After creating new record store. Write to record store failed with below error.

=======================================================================
2016-01-02 20:35:29,565 ERROR [MyStore-schema] [1415899750@jetty-7] com.endeca.itl.recordstore.impl.RecordStoreImpl: Error executing method RecordStoreImpl.writeRecords()
java.lang.NullPointerException
    at com.endeca.itl.recordstore.impl.RecordStoreImpl.handleInvalidRecord(RecordStoreImpl.java:282)
    at com.endeca.itl.recordstore.impl.RecordStoreImpl.validateRecords(RecordStoreImpl.java:274)
    at com.endeca.itl.recordstore.impl.RecordStoreImpl.writeRecords(RecordStoreImpl.java:242)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
    at com.endeca.itl.service.ServicePublisher$2.performInvocation(ServicePublisher.java:106)
    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
    at org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.invoke(AbstractJAXWSMethodInvoker.java:178)
    at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:64)
    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:75)
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:107)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:122)
    at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:211)
    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:213)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:193)
    at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:129)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:187)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:110)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:166)
    at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.endeca.itl.plugin.ThreadContextRunner$1$1.doWork(ThreadContextRunner.java:19)
    at com.endeca.itl.plugin.ThreadContextRunner.run(ThreadContextRunner.java:110)
    at com.endeca.itl.plugin.ThreadContextRunner$1.invoke(ThreadContextRunner.java:25)
    at $Proxy1.service(Unknown Source)
    at com.endeca.itl.bootstrap.CasBootStrapServlet.service(CasBootStrapServlet.java:105)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
    at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2016-01-02 20:35:29,566 WARN [] [1415899750@jetty-7] org.apache.cxf.phase.PhaseInterceptorChain: Application {http://recordstore.itl.endeca.com/}RecordStoreService#{http://recordstore.itl.endeca.com/}writeRecords has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault
    at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162)
    at org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.createFault(AbstractJAXWSMethodInvoker.java:213)
    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128)
    at org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.invoke(AbstractJAXWSMethodInvoker.java:178)
    at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:64)
    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:75)
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:107)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:122)
    at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:211)
    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:213)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:193)
    at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:129)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:187)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:110)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:166)
    at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.endeca.itl.plugin.ThreadContextRunner$1$1.doWork(ThreadContextRunner.java:19)
    at com.endeca.itl.plugin.ThreadContextRunner.run(ThreadContextRunner.java:110)
    at com.endeca.itl.plugin.ThreadContextRunner$1.invoke(ThreadContextRunner.java:25)
    at $Proxy1.service(Unknown Source)
    at com.endeca.itl.bootstrap.CasBootStrapServlet.service(CasBootStrapServlet.java:105)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
    at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.NullPointerException
    at com.endeca.itl.recordstore.impl.RecordStoreImpl.handleInvalidRecord(RecordStoreImpl.java:282)
    at com.endeca.itl.recordstore.impl.RecordStoreImpl.validateRecords(RecordStoreImpl.java:274)
    at com.endeca.itl.recordstore.impl.RecordStoreImpl.writeRecords(RecordStoreImpl.java:242)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
    at com.endeca.itl.service.ServicePublisher$2.performInvocation(ServicePublisher.java:106)
    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
    ... 44 more
=======================================================================

Solution : update record store configuration to handle(ignore) invalid records.

Step 1. Get record store configuration

recordstore-cmd.bat get-configuration -a MyStore-schema -f schema.xml

Step 2. Update recoed store configuration file with below line [edit xml file created in above step].

 <ignoreInvalidRecords>true</ignoreInvalidRecords>

Step 3. Set record store configuration using below command.

recordstore-cmd.bat set-configuration -a MyStore-schema -f schema.xml

Endeca Application Initialization error

After creating new endeca application successfully. Endeca application initialization failed with below error.
=========================================================================

SEVERE: Caught an exception while invoking method 'run' on object 'InitialSetup'. Releasing locks.

java.lang.reflect.InvocationTargetException

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:606)

  at com.endeca.soleng.eac.toolkit.Controller.invokeRequestedMethod(Controller.java:931)

  at com.endeca.soleng.eac.toolkit.Controller.execute(Controller.java:269)

  at com.endeca.soleng.eac.toolkit.Controller.main(Controller.java:137)

Caused by: com.endeca.soleng.eac.toolkit.exception.AppControlException: Unknown error executing a BeanShell script.

  at com.endeca.soleng.eac.toolkit.script.Script.runBeanShellScript(Script.java:184)

  at com.endeca.soleng.eac.toolkit.script.Script.run(Script.java:127)

  ... 7 more

Caused by: Sourced file: inline evaluation of: ``             IFCR.provisionSite();     CAS.importDimensionValueIdMappings("Store-d . . . '' : Error in method invocation: Method importDimensionValueIdMappings( java.lang.String, java.lang.String ) not found in class'com.endeca.soleng.eac.toolkit.component.CustomComponent' : at Line: 4 : in file: inline evaluation of: ``             IFCR.provisionSite();     CAS.importDimensionValueIdMappings("Store-d . . . '' : CAS .importDimensionValueIdMappings ( "Store-dimension-value-id-manager" , InitialSetup .getWorkingDir ( ) + "/test_data/initial_dval_id_mappings.csv" )

  at bsh.BSHMethodInvocation.eval(Unknown Source)

  at bsh.BSHPrimaryExpression.eval(Unknown Source)

  at bsh.BSHPrimaryExpression.eval(Unknown Source)

=========================================================================

Below is the error in application log file (MyStore.0.0.log) 

WARNING: Error parsing 'class' attribute:com.endeca.eac.toolkit.component.cas.ContentAcquisitionServerComponent

This attribute is in CAS custom component definition (DataIngest.xml).

Solution :  Add casStubs.jar from CAS_ROOT\lib\cas-dt\ to classpath in runcommand script.

Below is example from runcomman.bat file.

set CLASSPATH=%CLASSPATH%;C:\Endeca\CAS\11.1.0\lib\cas-dt\casStubs.jar