December 16, 2016

Useful XPath

Join Element Text

Input :
<employees>
    <employee>
        <empId>123</empId>
    </employee>
    <employee>
        <empId>456</empId>
    </employee>
</employees>

XPath : string-join(//empId, ',')

Output :
123,456

Distinct Values in XPath 1.0
Distinct function is not available in XPath 2.0. Therefore in XPath 1.0 you have to use the following.

Input :
<students>
<student>
<studentId>100</studentId>
<subject>Maths</subject>
</student>
<student>
<studentId>102</studentId>
<subject>Science</subject>
</student>
<student>
<studentId>101</studentId>
<subject>English</subject>
</student>
<student>
<studentId>100</studentId>
<subject>English</subject>
</student>
<student>
<studentId>100</studentId>
<subject>Science</subject>
</student>
</students>

XPath : //student/studentId[not(.=preceding::*)]

Output :
<result>
   <studentId>100</studentId>
   <studentId>102</studentId>
   <studentId>101</studentId>
</result>

December 2, 2016

XSLT Tips and Tricks

Remove CDATA section from a XML

Input XML
<request><![CDATA[ 
   <user>
     <firstname>Maheeka</firstname>
     <lastname>Jayasuriya</lastname>
   </user>]]>
</request>

XSLT
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
   <xsl:output indent="yes"/>
   <xsl:template match="@*|node()">
      <xsl:copy>
         <xsl:apply-templates select="@*|node()"/>
      </xsl:copy>
   </xsl:template>
   <xsl:template match="*:request/text()">
      <xsl:value-of disable-output-escaping="yes" select="."/>
   </xsl:template>
</xsl:stylesheet>

Transformed XML
<request> 
   <user>
      <firstname>Maheeka</firstname>
      <lastname>Jayasuriya</lastname>
   </user>
</request>

Extract Element Names to Element Values

Input XML
<user>
   <FirstName>Maheeka</FirstName>
   <LastName>Jayasuriya</LastName>
   <Age>26</Age>
</user>

XSLT
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
   <xsl:template match="//user">
      <user>
         <xsl:for-each select="*">
             <xsl:value-of select="local-name()"/> = <xsl:value-of select="."/>,  
         </xsl:for-each>
      </user>
   </xsl:template>
</xsl:stylesheet>

Transformed XML
<user>FirstName = Maheeka,  
LastName = Jayasuriya,  
Age = 26,  
</user>

Group Payload By a Distinct Value

Input XML
<students>
    <student>
        <studentId>100</studentId>
        <subject>Maths</subject>
    </student>
    <student>
        <studentId>102</studentId>
        <subject>Science</subject>
    </student>
    <student>
        <studentId>101</studentId>
        <subject>English</subject>
    </student>
    <student>
        <studentId>100</studentId>
        <subject>English</subject>
    </student>
    <student>
        <studentId>100</studentId>
        <subject>Science</subject>
    </student>
</students>

XSLT
<xsl:stylesheet version="1.0" xmlns:fn="http://www.w3.org/2005/xpath-functions" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:template match="/">
        <students>
            <xsl:for-each select="//student/studentId[not(.=preceding::*)]">
                <xsl:variable name="studentId" select="."/>
                <student>
                    <studentId>
                        <xsl:value-of select="$studentId"/>
                    </studentId>
                    <xsl:for-each select="//student[studentId=$studentId]/subject">
                        <subject>
                            <xsl:value-of select="."/>
                        </subject>
                    </xsl:for-each>
                </student>
            </xsl:for-each>
        </students>
    </xsl:template>
</xsl:stylesheet>

Transformed XML
<students xmlns:fn="http://www.w3.org/2005/xpath-functions">
    <student>
        <studentId>100</studentId>
        <subject>Maths</subject>
        <subject>English</subject>
        <subject>Science</subject>
    </student>
    <student>
        <studentId>102</studentId>
        <subject>Science</subject>
    </student>
    <student>
        <studentId>101</studentId>
        <subject>English</subject>
    </student>
</students>

November 24, 2016

Parasoft Virtualizer Tips for Generating Dynamic Responses

This post explains how to create dynamic responses when creating Virtual Assets in Parasoft Virtualizer. The dynamic responses may be created depending on the request payload and headers. 

We will work on three cases of dynamic responses here.
  1. Dynamic response from request payload
  2. Dynamic response from request payload with CDATA
  3. Dynamic response from request headers

Getting Started

Let us start by first creating a new Virtual Asset 'Empty' project


Right click on Responder Suite and Add Responder and select JSON Message Responder or Plain XML Message Responder as required. For the purpose of this post, I am selecting Plain XML Message Responder.

In this post we will be providing a firstname and lastname and reply with a email address constructed as "<firstname>.<lastname>@company.com"

1. Dynamic Response from Request Payload

This is the most simple form of dynamic response. We extract data from the request to payload to construct the response. 

Request : 
<request>
<user>
<firstname>Maheeka</firstname>
<lastname>Jayasuriya</lastname>
    </user>
</request>

Expected Response : 
<user>
    <email>Maheeka.Jayasuriya@maheeka.me</email>
</user>

In the created new Responder, go to Options tab and copy the request to the Request Template as below in the Literal view. 


Now go to the Response tab and copy the expected response in the Literal view.

If you now switch to the Form XML response, you are able to see the response and its values of the elements as below.


In order to get the 'firstname' and 'lastname' from the request, we need to extract the data by using a XML Data Bank

For this, right click on the Responder > Add Output.
Select Incoming Request > Payload > XML Data Bank
Now you should be able to see the request we configured earlier on the left view. Select 'firstname' and click on 'Extract Element'. This will add a Data Source Column with the value extracted from the selected 'firstname' element's XPath.

Select the added Data Source and click 'Modify'. Click on 'Evaluate XPath' and you should be able to see the extracted value from the request. Here, 'firstname' element that was selected resolves to 'Maheeka'


However, since we need to create the email from 'firstname' and 'lastname', let's modify this entry as below. You can add any XPath using XPath 1.0 functions for this. I am modifying the XPath as 'concat(/request/user[1]/firstname[1]/text(), '.' , /request/user[1]/lastname[1]/text(), '@maheeka.me')' which resolves to 'Maheeka.Jayasuriya@maheeka.me'
Also, we need to modify the Data Source Column name to email.
Now I need to create the response to get the email from the Data Source Column that we just created.

Go back to the Responder's Response tab and click on email element and select 'Parameterized' for Value. This will list down the columns that we created and you can select 'email' from here. 
Now you need to add this Virtual Asset to a local server or a remote server and invoke the service with a POST request as below, 

2. Dynamic Response from Request Payload with CDATA

Here we get a part of the response in a CDATA section. We need to extract data from this CDATA section to construct the response. 

Request : 
<request>
<![CDATA[ 
<user>
<firstname>Maheeka</firstname>
<lastname>Jayasuriya</lastname>
</user>
]]>
</request>

Expected Response : 
<user>
    <email>Maheeka.Jayasuriya@maheeka.me</email>
</user>

Most of the steps are similar to above. However, in addition, we have a step to extract data from the CDATA section. In order to do that, we are adding an XSL transformation before we extract data to the XML Data Bank.

Right click on the Responder > Add Output  > Payload and Select XSLT.

Following XSLT can be used to extract the CDATA from the XML payload.

<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output indent="yes"/>
<xsl:template match="@*|node()">
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
</xsl:copy>
</xsl:template>
<xsl:template match="*:request/text()">
<xsl:value-of disable-output-escaping="yes" select="."/>
</xsl:template>
</xsl:stylesheet>

If you test out this XSLT with the above payload, you will see that this transforms the request to below. You can easily test this using an online tool like http://www.freeformatter.com/xsl-transformer.html (there are many other tools as well).

<request>
<user>
<firstname>Maheeka</firstname>
<lastname>Jayasuriya</lastname>
</user>
</request>

Now, this is similar to the payload earlier in the 1st scenario. Therefore, we can do the same steps to create an XML Data Bank to extract the first name and last name and set the email Data Source column and set it to the response.

Right click on the Incoming Request -> XSLT and Add Output > Transformed Source > XML Data Bank. Add the converted payload from XSLT to the left panel and extract from their XPath to create email column as below.



Now you need to add the email column as the parameter for email element in the response payload. Redeploy the virtual asset and invoke the service similar to, the earlier scenario. Following is the response from invoking the service using Postman.

3. Dynamic response from request headers

Sometimes, the request data are coming as headers or as URL parameters. At this point, we need to extract the data from the header to get the 'firstname' and 'lastname' to construct email.

Request : 
GET http://localhost:9080/DynamicResponderServer?firstname=Maheeka&lastname=Jayasuriya

Expected Response : 
<user>
    <email>Maheeka.Jayasuriya@maheeka.me</email>
</user>

First, few steps are similar to above, up until creating a Responder. Here we need to extract the query parameters from the raw request. Therefore, we will be using a Text Data Bank for this.

Right Click on the Responder and Add Output > Incoming Request > Transport Header > Text Data Bank.


Go to Postman and add query parameters 'firstname' and 'lastname' as below.
If you click on 'Code' link on the bottom right corner as in above diagram, you are able to copy the raw request from Postman as below.
Following is the raw request for this sample.

GET /DynamicResponderServer?firstname=Maheeka&amp;lastname=Jayasuriya HTTP/1.1
Host: localhost:9080
Content-Type: application/xml
Cache-Control: no-cache
Postman-Token: 9cd3e89e-74d1-8b78-ac5e-c7ccdd4b7e0e

Now go back to Virtualizer and add this raw request on the Text Content panel of the Text Data Bank.
By selecting any text you need from the raw request, you can add a Data Source Column.

First, let's try to get the full URL of the request. Select the URL from the first line in the request as below, and click on Add.



Create an additional element in the response payload (refer above scenario steps on constructing the response format) called url and select the URL Data Source column here for the value as below.

Now deploy the Virtual Asset and invoke via Postman.
If you take a look at the URL that is available in the response, you will notice that the query parameters are not in the order we sent in the request.

In the request we sent as :
/DynamicResponderServer?firstname=Maheeka&lastname=Jayasuriya

In Virtualizer however, the request have come as :
/DynamicResponderServer?lastname=Jayasuriya&amp;firstname=Maheeka

This is something I had to figure out the hard way :). I am not sure why this happens in Virtualizer. However, it is best to first try to get the value of the URL like this and then do the 'firstname' and 'lastname' extractions.

Now back to the Text Data Bank view. We need to modify the URL in the raw request with the one we got by invoking via Postman.

Now Select the 'Jayasuriya' part and Click on Add > OK

The parameters are listed in the table underneath the text panel. If you click on Modify after selecting 'lastname', you will see the following Dialog. Here Virtualizer has extracted few characters before and after the text that we selected, in order to uniquely identify the value. 

'
We can make it further unique by adding the complete query parameter name in the 'Left Hand text as below. Also, we need to select URL Decoded to allow special characters to be decoded when extracting. For example & becomes &amp; when decoded. Since query parameters will have & sign, we need to select this option.

We need to do the same with 'firstname' as in the Dialog below.


Now let's go back to the response view and remove the previously added URL element. Now if you select on the email element, we can now add the email address value as below. We do not have a column as email this time. Instead, we are constructing the email here itself in the Fixed option as "${firstname}.${lastname}@maheeka.me". ${column_name} is the notation used to extract values from a data column.


Another way of constructing the email here is to use a JavaScript code. If you want to use a JavaScrip function, instead of Fixed or Parameterized, select Script option.
Click on 'Edit Script'. This will bring up the below Dialog, where you can write a JavaScript function to do the same construction of email address. If there are no errors in the JavaScript source, if you click on Evaluate there should be no errors and the Methods drop down should be populated. Select the relevant method (if you have multiple) that will be executed to get the value. 

Following is the JavaScript function : 

function getEmailAddress(context) {
   return context.getValue("Text Data Bank", "firstname") + "." + context.getValue("Text Data Bank", "lastname") + "@maheeka.me"
}

context.getValue("Text Data Bank", "firstname")  code segment means to get the column 'firstname' from the data bank named 'Text Data Bank'. If you renamed the data bank, modify this code accordingly.

Now we can make a request to this service using Postman as below.

We did a sample of extracting query parameters here. However, we can extract any header using the same steps.

November 19, 2016

LDAP User Authentication

What is LDAP and LDAP Authentication ?

To get started on what and how LDAP works, take a look at [1].

Quoting from the above article, following is about how LDAP Authentication works :
To perform any of these LDAP operations, an LDAP client needs to establish a connection with an LDAP server. The LDAP protocol specifies the use of TCP/IP port number 389, although servers may run on other ports.
The LDAP protocol also defines a simple method for authentication. LDAP servers can be set up to restrict permissions to the directory. Before an LDAP client can perform an operation on an LDAP server, the client must authenticate itself to the server by supplying a distinguished name and password. If the user identified by the distinguished name does not have permission to perform the operation, the server does not execute the operation.
In order write a simple LDAP Authenticator using Java, refer to the article [2] first to get an idea.

As per the above quote and explanations, in order to perform an LDAP based user authentication, we validate the user's permission by trying to execute an operation on LDAP.

Java LDAP Authenticator

import javax.naming.Context;
import javax.naming.directory.DirContext;
import javax.naming.directory.InitialDirContext;
import java.util.Hashtable;
/**
 * Open a connection to the LDAP server with uid and password and authenticate the user.
 */
public class LDAPAuthenticator {
    private static String UID_FORMAT_STRING = "uid=%1s,%2s";
    private static String VDS_SERVER = "LDAPS://localhost:389";
    private static final String PEOPLE_OU = "ou=people,dc=local,dc=com";
    public static void main(String[] args) {
        Hashtable<String, String> authEnv = new Hashtable<String, String>();
        authEnv.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
        authEnv.put(Context.PROVIDER_URL, VDS_SERVER);
        authEnv.put(Context.SECURITY_AUTHENTICATION, "simple"); //default authentication
        authEnv.put(Context.SECURITY_PRINCIPAL, String.format(UID_FORMAT_STRING, "uidName", PEOPLE_OU));
        authEnv.put(Context.SECURITY_CREDENTIALS, "password");
        try {
            DirContext ctx = new InitialDirContext(authEnv);
            ctx.close();
            System.out.println("User Authentication Successful");
        } catch (Exception e) {
            System.out.println("User Authentication Failed");
            e.printStackTrace();
        }
    }
}

In addition if you need perform attribute matching along with authentication use the following code as an example :

Attributes matchAttrs = new BasicAttributes(true); // ignore attribute name case
matchAttrs.put(new BasicAttribute("uid", "AARSupport"));

// Search for objects with those matching attributes
NamingEnumeration answer = ctx.search(PEOPLE_OU, matchAttrs);

WSO2 LDAP Connector

If you are using WSO2 ESB, you can use the WSO2 LDAP connector to perform LDAP authentication. You can download the connector from [6]. Documentation on using the LDAP connector for authentication can be found at [7].

Use the following configurations to call the Authenticate operation.

Create a local entry with authentication details as below.

<ldap.init xmlns="http://ws.apache.org/ns/synapse">
      <providerurl>LDAPS://localhost:389</providerurl>
      <securityprincipal>uid=UIDName,ou=people,dc=local,dc=com</securityprincipal>
      <securitycredentials>password</securitycredentials>
      <secureconnection>false</secureconnection>
      <disablesslcertificatechecking>false</disablesslcertificatechecking>
</ldap.init>


Or you can also create local entry as below without authentication details since they are again later provided at the authenticate operation.

<ldap.init xmlns="http://ws.apache.org/ns/synapse">
      <providerurl>LDAPS://localhost:389</providerurl>
      <securityprincipal></securityprincipal>
      <securitycredentials></securitycredentials>
      <secureconnection>false</secureconnection>
      <disablesslcertificatechecking>false</disablesslcertificatechecking>
</ldap.init>

In the mediation, you can call the LDAP connector's authenticate operation to perform authentication.

<ldap.authenticate configkey="LDAPConfig">
        <dn>uid=UIDName,ou=people,dc=local,dc=com</dn>
        <password>password</password>
</ldap.authenticate>


If the authentication is successful, you would be getting a Success response back.

References : 

  1. https://docs.oracle.com/cd/E19957-01/816-6402-10/ldap.htm
  2. http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ldap.html
  3. https://tools.ietf.org/html/rfc4519#section-2.39
  4. http://stackoverflow.com/questions/2522770/how-to-check-user-password-in-ldap-whith-java-with-given-ldapcontext
  5. http://stackoverflow.com/questions/7813868/whats-the-difference-in-using-distinguished-name-with-cn-or-uid-when-logging-in
  6. https://store.wso2.com/store/assets/esbconnector/details/4ecf8dde-60f3-4e91-ba22-5f49a4e302f4
  7. https://docs.wso2.com/display/ESBCONNECTORS/Working+with+User+Authentication+in+LDAP

Import Certificate to IntelliJ IDEA to Runtime


In order to import a certificate to run-time in IntelliJ IDEA do the following steps.

1. Go to Run > Edit Configuration


2. In VM options provide the trustStore as below : 
-Djavax.net.ssl.trustStore=C:\Source\vds\vdsstore.jks -Djavax.net.ssl.trustStorePassword=vdsstore


In order to import the certificate to a trust store do the following : 

keytool -import -alias vdsfmr -file Cert.cer -keystore wso2carbon.jks
VMArgument  -Djavax.net.ssl.trustStore=vdsStore -Djavax.net.ssl.trustStorePassword=vdsStore

Reference : https://docs.oracle.com/javase/tutorial/security/toolsign/rstep2.html 

TroubleShooting

Unable to Find Cert :

Ceritificate is not available during runtime. To resolve add the VM Arguments pointing to Trust Store as above.

javax.naming.CommunicationException: simple bind failed: localhost:389 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]
at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:218)
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2740)
at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:316)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:193)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:211)
at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:154)
at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:84)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
at javax.naming.InitialContext.init(InitialContext.java:242)
at javax.naming.InitialContext.<init>(InitialContext.java:216)
at javax.naming.directory.InitialDirContext.<init>(InitialDirContext.java:101)
at LDAPAuthenticator.main(LDAPAuthenticator.java:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1904)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:279)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:273)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1446)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:209)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:913)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:849)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1023)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:889)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at com.sun.jndi.ldap.Connection.run(Connection.java:855)
at java.lang.Thread.run(Thread.java:745)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1428)
... 12 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
... 18 more


When the Trust Store is Not Provided as an Absolute Path : 
Reference : http://stackoverflow.com/questions/4764611/java-security-invalidalgorithmparameterexception-the-trustanchors-parameter-mus

I was able to solve this error by providing the absolute path to the trust store instead of the relative path.

javax.naming.CommunicationException: simple bind failed: localhost:389 [Root exception is javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty]
 at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:218)
 at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2740)
 at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:316)
 at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:193)
 at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:211)
 at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:154)
 at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:84)
 at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
 at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
 at javax.naming.InitialContext.init(InitialContext.java:242)
 at javax.naming.InitialContext.<init>(InitialContext.java:216)
 at javax.naming.directory.InitialDirContext.<init>(InitialDirContext.java:101)
 at VDSAuth.main(VDSAuth.java:31)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
 at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1904)
 at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1862)
 at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1845)
 at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1771)
 at sun.security.ssl.AppInputStream.read(AppInputStream.java:113)
 at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
 at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
 at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
 at com.sun.jndi.ldap.Connection.run(Connection.java:855)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
 at sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:90)
 at sun.security.validator.Validator.getInstance(Validator.java:179)
 at sun.security.ssl.X509TrustManagerImpl.getValidator(X509TrustManagerImpl.java:314)
 at sun.security.ssl.X509TrustManagerImpl.checkTrustedInit(X509TrustManagerImpl.java:173)
 at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:186)
 at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
 at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1428)
 at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:209)
 at sun.security.ssl.Handshaker.processLoop(Handshaker.java:913)
 at sun.security.ssl.Handshaker.process_record(Handshaker.java:849)
 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1023)
 at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:889)
 at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
 ... 5 more
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
 at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
 at java.security.cert.PKIXParameters.<init>(PKIXParameters.java:120)
 at java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:104)
 at sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:88)
 ... 18 more

October 30, 2016

Mocking Endpoint Behaviours for Troubleshooting in ESB


This article explains some common timeout and latency scenarios in endpoints and information on troubleshooting these issues. Following three scenarios will be discussed.
  1. Backend responds as expected but response time is fairly low
  2. Backend timeout
  3. 101503 Error : Backend connection refused
  4. Unknown Host exception
In order to explain the different scenarios, the easiest approach is to create a mock service for troubleshooting. For this article I am creating a mock service with SOAPUI. You may use any other tool to do the same. Refer [1] on more details for creating mock services using SOAPUI.

Once we create a mock service it is accessible with http://localhost:8080/mockservice.

Case 1 : Endpoint Responds with Low Response Time

This case is fairly straightforward. First we need to create the mock service and point to the mock service url in the ESB endpoint definition. Usually the cases to verify are the behaviour at different responses. To simulate this we can create multiple responses and attach to the mock service. This allows us to validate the behaviour for different responses.


Case 2 : Endpoint Timeout

Refer below endpoint timeout duration in the endpoint definition. We have set it to be 3000ms.


<?xml version="1.0" encoding="UTF-8"?>
<endpoint name="MockServiceEndpoint" xmlns="http://ws.apache.org/ns/synapse">
    <http method="post" uri-template="http//localhost:8080/mockservice">
        <timeout>
            <duration>3000</duration>
            <responseAction>fault</responseAction>
        </timeout>
        <suspendOnFailure>
            <errorCodes>-1</errorCodes>
            <initialDuration>0</initialDuration>
            <progressionFactor>1.0</progressionFactor>
            <maximumDuration>0</maximumDuration>
        </suspendOnFailure>
        <markForSuspension>
            <errorCodes>-1</errorCodes>
        </markForSuspension>
    </http>
</endpoint>

In this case, since endpoint timeout duration is 3000ms, we can verify four cases where;

  1. No backend latency
  2. 2000 < timeout_duration <  3000 (latency close to 3000)
  3. timeout_duration  > 4000 (latency much higher than 3000)
  4. timeout_duration = 3000

From above four cases, 3 and 4 cases should cause endpoint to timeout. Other scenarios should respond and the rest of the mediation flow should work correct. In order to add the response latency to mock service, add a sleep command on the onRequest script of the mock service window as below.



Notice the delay in response time for the request that has gone past 3000ms in the below screenshot (left bottom)


By changing the sleep duration we can test out the behaviour of response time at the above four scenarios listed above.

Case 3 : 101503 - Endpoint Connection Refused

When faced with this error, a latency of ~10000 ms will be observed from the backend. Considering the fact that the endpoint timeout was 3000ms and it seemed highly improbable to exceed this number.

In order to mock this scenario, you need to host the mock service in a different machine in the same network and point to it as the endpoint (referred to as the mock server hereafter). While requests were sent out, the ‘mock server’ was disconnected from the network and ~10000ms of latency was observed. When ‘mock server’ is reconnected to the network normal behaviour is observed again.

To explain further the latency of ~10000ms is caused by connection refused error where a hostname is valid by DNS but is no longer available. This is why the ‘mock server’ had to be disconnected from the network to mock this behaviour. This is different from Case 4 described below.

Another way to mock this behaviour is to use an IP address which timeouts when pinged as the hostname in the endpoint definition.

Explanation on the Observation from WSO2 ESB perspective

In ESB, whenever a request is made a callback is registered for the request. This callback is responsible for sending back the response and doing the required processing. When backend does not response, these callbacks have to be cleared. Clearing is done by a task (TimeoutHandler) which runs every 15000 ms (by default).  This time duration can be controlled by overriding the default time duration with “synapse.timeout_handler_interval” property in ‘synapse.properties’ file.

To further elaborate, what happens here is that TimeoutHandler is executed every 15s and the callbacks get cleared within a 15s timeframe. Thus the response time will be somewhere below 15s, yet above 3s (to allow initial endpoint timeout duration). Attached graph of response time at default timeout for the mock API we created further explains this fact.


Response time will go to a lower range when we reduce the timeout_handler_interval. However, this means a higher frequency of timeout handler task being executed and will in turn impact performance.

To confirm the above description please refer the following observations when timeout_handler_interval is reduced.

Timeout_handler_interval = 5000ms

Timeout_handler_interval = 1000ms

Case 4 : Unknown Host Exception

Another scenario that was verified was giving an invalid host name, which resulted in an ‘Unknown Host Exception’, but this is handled within the 3000ms timespan.

Apart from these, there are a number of scenarios related to troubleshooting issues in endpoints. Refer for possible error scenarios in endpoints in [2]

References

[1] https://www.soapui.org/soap-mocking/getting-started.html
[2] https://docs.wso2.com/display/ESB480/Error+Handling#ErrorHandling-codes

August 8, 2016

Using SAML SSO Authorize Carbon Admin Services from Jaggery App

To give a background on the above scenario, please refer the following diagram.



A Jaggery App is deployed on WSO2 Application Server (AS) and we have enabled SAML 2.0 based SSO for the Jaggery app using WSO2 Identity Server (IS) . You can find more details on enabling SSO with IS at [1]. ESB also shares the same IDP.

What we are doing here is authenticating the Jaggery App from IDP by sending a SAML request and using the same SAML response to authenticate ESB and get a session cookie. This cookie can then be used to invoke admin services of ESB.

[JAGGERY_APP]/jagg/jaggery_acs file acts as the assertion consumer service (ACS). This is the same that is configured in the IDP. Once the request comes to jaggery_acs.jag we will validate the SAML response and get the cookie from ESB.

Once we have the SAML response, use the following code to authenticate ESB.

 //authenticate ESB
var ws = require("ws");
var requestESB = new ws.WSRequest();
var options = new Array();
options.useSOAP = 1.2;
options.action = "urn:login";
var endPoint = "https://localhost:9453/services/SAML2SSOAuthenticationService";
var payload = '<sso:login xmlns:sso="http://sso.saml2.authenticator.identity.carbon.wso2.org"><sso:authDto><xsd:response xmlns:xsd="http://dto.sso.saml2.authenticator.identity.carbon.wso2.org/xsd">' + samlResponse + '</xsd:response></sso:authDto></sso:login>';
requestESB.open(options,endPoint, false);
requestESB.send(payload);
var responseESB = requestESB.responseE4X;
var adminSession = requestESB.getResponseHeader("Set-Cookie");
session.put("esb-auth-cookie", adminSession);


Use the following code segment to invoke ESB Admin services with the above cookie

var restApiAdminUrl = site.esb.serverBaseURL + "t/" + tenantDomain + "/services/" + REST_API_ADMIN_SERVICE + "/";
var requestPayload = "<xsd:getAPIsForListing xmlns:xsd=\"http://org.apache.axis2/xsd\">"
                      +"  <xsd:pageNumber>0</xsd:pageNumber>"
                      +"  <xsd:itemsPerPage>100</xsd:itemsPerPage>"
                      +"</xsd:getAPIsForListing>";

var ws = require("ws");
var request = new ws.WSRequest();
var options = new Array();
options.useSOAP = 1.2;
options.action = "urn:getAPIsForListing";
options["HTTPHeaders"] = [{name: "cookie", value: session.get("esb-auth-cookie")}];
request.open(options, restApiAdminUrl, false);
request.send(requestPayload);
var response = request.responseE4X;



References

[1] https://docs.wso2.com/display/IS500/Configuring+Single+Sign-On+with+SAML+2.0
[2] http://wso2.com/library/articles/2016/02/article-how-to-setup-a-wso2-api-manager-store-login-with-google/

June 12, 2016

WSO2 Admin Services

WSO2 Admin Services

All WSO2 Products internally perform all of it's operations via admin services. From the management console, these admin services are called to facilitate the features.

You can find more information on this at : https://docs.wso2.com/display/Carbon420/Calling+Admin+Services+from+Apps

By default, admin services WSDLs are not exposed in a product. Therefore need to do the following to access the WSDLs.

  • Set the <HideAdminServiceWSDLs> element to false in <PRODUCT_HOME>/repository/conf/carbon.xml file and restart the server.
  • Additionally to access the OSGI console, start the server as <PRODUCT_HOME>/bin/wso2server.sh -DosgiConsole. By hitting enter on the console it will start the OSGI shell
  • osgi> listAdminServices will list all the admin services available in the product.


Invoking Admin Services

There are a few ways to invoke admin services.
  1. Invoke with SOAP UI using the WSDLs of the admin services
  2. Invoke via CURL
  3. Invoke programmatically via the service stubs.

Using SOAP UI to Invoke Admin Services

We can create a new SOAP UI project by using the WSDL. We can get the sample requests and fill up the necessary data to invoke the service operations.

Even if we are using CURL to invoke the admin services, we need to get the request format which is easiest to be done by importing the WSDL as a SOAP project to SOAP UI first.

Using CURL to Invoke Admin Services

As mentioned above, best approach is to get the request format from SOAPUI. Once you have the request format, we can invoke admin services as below.

1. First we need to authenticate. Therefore, invoke the AuthenticationAdmin service.

Request (auth.xml):

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://org.apache.synapse/xsd">
 <soapenv:Header/>
 <soapenv:Body>
      <aut:login xmlns:aut="http://authentication.services.core.carbon.wso2.org">
         <aut:username>admin</aut:username>
         <aut:password>admin</aut:password>
         <aut:remoteAddress>localhost</aut:remoteAddress>
      </aut:login>
   </soapenv:Body>
</soapenv:Envelope>

CURL command :

curl -k -v -H "SOAPAction: urn:login" -H "Content-Type: text/xml" -d @auth.xml https://localhost:9443/services/AuthenticationAdmin

Response :
*   Trying ::1...
* connect to ::1 port 9443 failed: Connection refused
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: localhost
> POST /services/AuthenticationAdmin HTTP/1.1
> Host: localhost:9443
> User-Agent: curl/7.43.0
> Accept: */*
> SOAPAction: urn:login
> Content-Type: text/xml
> Content-Length: 467
* upload completely sent off: 467 out of 467 bytes
< HTTP/1.1 200 OK
< Set-Cookie: JSESSIONID=E74FD5AB56A283CD261E26A77B9854FB; Path=/; Secure; HttpOnly
< Content-Type: text/xml;charset=UTF-8
< Transfer-Encoding: chunked
< Date: Sun, 12 Jun 2016 03:20:09 GMT
< Server: WSO2 Carbon Server
* Connection #0 to host localhost left intact

<?xml version='1.0' encoding='UTF-8'?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Body>
      <ns:loginResponse xmlns:ns="http://authentication.services.core.carbon.wso2.org">
         <ns:return>true</ns:return>
      </ns:loginResponse>
   </soapenv:Body>
</soapenv:Envelope>

At this time you should also be able to see the following command in the ESB terminal (or the relevant product's terminal)

[2016-06-12 08:52:25,931]  INFO - CarbonAuthenticationUtil 'admin@carbon.super [-1234]' logged in at [2016-06-12 08:52:25,931+0530]

2. From the response, get the JSessionID. This will be used for later requests to authenticate.

JSESSIONID=E74FD5AB56A283CD261E26A77B9854FB

3. If you are trying this with ESB, we can try invoking TemplateAdminService as below.

Request (template.xml):
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://org.apache.synapse/xsd">
   <soapenv:Header/>
   <soapenv:Body>
      <xsd:getTemplates>
         <!--Optional:-->
         <xsd:pageNumber>0</xsd:pageNumber>
         <!--Optional:-->
         <xsd:templatePerPage>200</xsd:templatePerPage>
      </xsd:getTemplates>
   </soapenv:Body>
</soapenv:Envelope>


CURL Command

curl -k -v -H "Cookie: JSESSIONID=7D8798EB0A39E18194E778461F7E3E6D" -H "SOAPAction: urn:getTemplates" -H "Content-Type: text/xml" -d @template.xml https://localhost:9443/services/TemplateAdminService

Response :
*   Trying ::1...
* connect to ::1 port 9443 failed: Connection refused
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: localhost
> POST /services/TemplateAdminService HTTP/1.1
> Host: localhost:9443
> User-Agent: curl/7.43.0
> Accept: */*
> Cookie: JSESSIONID=7D8798EB0A39E18194E778461F7E3E6D
> SOAPAction: urn:getTemplates
> Content-Type: text/xml
> Content-Length: 389
* upload completely sent off: 389 out of 389 bytes
< HTTP/1.1 200 OK
< Content-Type: text/xml;charset=UTF-8
< Transfer-Encoding: chunked
< Date: Sun, 12 Jun 2016 03:33:22 GMT
< Server: WSO2 Carbon Server
* Connection #0 to host localhost left intact

<?xml version='1.0' encoding='UTF-8'?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Body>
      <ns:getTemplatesResponse xmlns:ax2246="http://common.templates.mediation.carbon.wso2.org/xsd" xmlns:ns="http://org.apache.synapse/xsd">
         <ns:return xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ax2246:TemplateInfo">
            <ax2246:artifactContainerName xsi:nil="true"/>
            <ax2246:description xsi:nil="true"/>
            <ax2246:enableStatistics>false</ax2246:enableStatistics>
            <ax2246:enableTracing>false</ax2246:enableTracing>
            <ax2246:isEdited>false</ax2246:isEdited>
            <ax2246:name>getUserID</ax2246:name>
         </ns:return>
      </ns:getTemplatesResponse>
   </soapenv:Body>
</soapenv:Envelope>

Note : 

For all the requests, make sure to use the SOAP 1.1 which is xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" and not xmlns:soap="http://www.w3.org/2003/05/soap-envelope", since this will result in an error invoking the admin services with CURL as  : 

<?xml version='1.0' encoding='UTF-8'?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header xmlns:wsa="http://www.w3.org/2005/08/addressing">
      <wsa:Action>http://www.w3.org/2005/08/addressing/soap/fault</wsa:Action>
   </soapenv:Header>
   <soapenv:Body>
      <soapenv:Fault>
         <faultcode>soapenv:VersionMismatch</faultcode>
         <faultstring>Transport level information does not match with SOAP Message namespace URI</faultstring>
         <detail/>
      </soapenv:Fault>
   </soapenv:Body>
</soapenv:Envelope>

Using Service Stubs to Invoke Admin Services

Following is the code segment to invoke to get the templates from ESB server by using the TemplateAdminService stub. 

First step is to authenticate : 

TemplateAdminServiceStub templateAdminServiceStub = new TemplateAdminServiceStub("https://localhost:9443/services/TemplateAdminService");
ServiceClient client = templateAdminServiceStub._getServiceClient();
Options client_options = client.getOptions();
HttpTransportProperties.Authenticator authenticator = new HttpTransportProperties.Authenticator();
authenticator.setUsername("admin");
authenticator.setPassword("admin");
authenticator.setPreemptiveAuthentication(true);
client_options.setProperty(org.apache.axis2.transport.http.HTTPConstants.AUTHENTICATE, authenticator);
client.setOptions(client_options);

Next we can invoke the service operation : 

OMElement templateOMElement = templateAdminServiceStub.getTemplate();

Invoking Admin Services in Tenant Mode

To invoke admin services in tenant mode, we only need to change the admin service URL with tenant URL and the username and password to the tenant.

E.g. :
Admin Service URL :  https://localhost:9443/services/t/maheeka.com/TemplateAdminService
Username : admin@maheeka.com
Password : admin123

Troubleshooting

1. Error when invoking admin service

[2016-05-25 16:41:36,054] ERROR - Class System failure.null
java.lang.NullPointerException
at org.wso2.carbon.server.admin.module.handler.AuthorizationHandler.doAuthorization(AuthorizationHandler.java:104)
at org.wso2.carbon.server.admin.module.handler.AuthorizationHandler.invoke(AuthorizationHandler.java:87)
at org.apache.axis2.engine.Phase.invokeHandler(Phase.java:340)
at org.apache.axis2.engine.Phase.invoke(Phase.java:313)
at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:261)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:167)
at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:398)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:145)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2016-05-25 16:41:36,055] ERROR - AxisEngine System failure.
org.apache.axis2.AxisFault: System failure.
at org.wso2.carbon.server.admin.module.handler.AuthorizationHandler.invoke(AuthorizationHandler.java:93)
at org.apache.axis2.engine.Phase.invokeHandler(Phase.java:340)
at org.apache.axis2.engine.Phase.invoke(Phase.java:313)
at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:261)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:167)
at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:398)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:145)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2016-05-25 16:41:36,057] ERROR - ServerWorker Error processing POST request for : /services/CarbonAppUploader
org.apache.axis2.AxisFault: System failure.
at org.wso2.carbon.server.admin.module.handler.AuthorizationHandler.invoke(AuthorizationHandler.java:93)
at org.apache.axis2.engine.Phase.invokeHandler(Phase.java:340)
at org.apache.axis2.engine.Phase.invoke(Phase.java:313)
at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:261)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:167)
at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:398)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:145)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Solution : 
ESB or APIM has two types of transports - servlet and passthrough. The admin services are exposed through servlet transport. Therefore, we need to invoke admin services through transport port. E.g. : https://localhost:9443/services/AuthenticationAdmin. The common reason for this error is that the admin service WSDLs are exposed on passthrough port (8243).