October 30, 2016

Mocking Endpoint Behaviours for Troubleshooting in ESB


This article explains some common timeout and latency scenarios in endpoints and information on troubleshooting these issues. Following three scenarios will be discussed.
  1. Backend responds as expected but response time is fairly low
  2. Backend timeout
  3. 101503 Error : Backend connection refused
  4. Unknown Host exception
In order to explain the different scenarios, the easiest approach is to create a mock service for troubleshooting. For this article I am creating a mock service with SOAPUI. You may use any other tool to do the same. Refer [1] on more details for creating mock services using SOAPUI.

Once we create a mock service it is accessible with http://localhost:8080/mockservice.

Case 1 : Endpoint Responds with Low Response Time

This case is fairly straightforward. First we need to create the mock service and point to the mock service url in the ESB endpoint definition. Usually the cases to verify are the behaviour at different responses. To simulate this we can create multiple responses and attach to the mock service. This allows us to validate the behaviour for different responses.


Case 2 : Endpoint Timeout

Refer below endpoint timeout duration in the endpoint definition. We have set it to be 3000ms.


<?xml version="1.0" encoding="UTF-8"?>
<endpoint name="MockServiceEndpoint" xmlns="http://ws.apache.org/ns/synapse">
    <http method="post" uri-template="http//localhost:8080/mockservice">
        <timeout>
            <duration>3000</duration>
            <responseAction>fault</responseAction>
        </timeout>
        <suspendOnFailure>
            <errorCodes>-1</errorCodes>
            <initialDuration>0</initialDuration>
            <progressionFactor>1.0</progressionFactor>
            <maximumDuration>0</maximumDuration>
        </suspendOnFailure>
        <markForSuspension>
            <errorCodes>-1</errorCodes>
        </markForSuspension>
    </http>
</endpoint>

In this case, since endpoint timeout duration is 3000ms, we can verify four cases where;

  1. No backend latency
  2. 2000 < timeout_duration <  3000 (latency close to 3000)
  3. timeout_duration  > 4000 (latency much higher than 3000)
  4. timeout_duration = 3000

From above four cases, 3 and 4 cases should cause endpoint to timeout. Other scenarios should respond and the rest of the mediation flow should work correct. In order to add the response latency to mock service, add a sleep command on the onRequest script of the mock service window as below.



Notice the delay in response time for the request that has gone past 3000ms in the below screenshot (left bottom)


By changing the sleep duration we can test out the behaviour of response time at the above four scenarios listed above.

Case 3 : 101503 - Endpoint Connection Refused

When faced with this error, a latency of ~10000 ms will be observed from the backend. Considering the fact that the endpoint timeout was 3000ms and it seemed highly improbable to exceed this number.

In order to mock this scenario, you need to host the mock service in a different machine in the same network and point to it as the endpoint (referred to as the mock server hereafter). While requests were sent out, the ‘mock server’ was disconnected from the network and ~10000ms of latency was observed. When ‘mock server’ is reconnected to the network normal behaviour is observed again.

To explain further the latency of ~10000ms is caused by connection refused error where a hostname is valid by DNS but is no longer available. This is why the ‘mock server’ had to be disconnected from the network to mock this behaviour. This is different from Case 4 described below.

Another way to mock this behaviour is to use an IP address which timeouts when pinged as the hostname in the endpoint definition.

Explanation on the Observation from WSO2 ESB perspective

In ESB, whenever a request is made a callback is registered for the request. This callback is responsible for sending back the response and doing the required processing. When backend does not response, these callbacks have to be cleared. Clearing is done by a task (TimeoutHandler) which runs every 15000 ms (by default).  This time duration can be controlled by overriding the default time duration with “synapse.timeout_handler_interval” property in ‘synapse.properties’ file.

To further elaborate, what happens here is that TimeoutHandler is executed every 15s and the callbacks get cleared within a 15s timeframe. Thus the response time will be somewhere below 15s, yet above 3s (to allow initial endpoint timeout duration). Attached graph of response time at default timeout for the mock API we created further explains this fact.


Response time will go to a lower range when we reduce the timeout_handler_interval. However, this means a higher frequency of timeout handler task being executed and will in turn impact performance.

To confirm the above description please refer the following observations when timeout_handler_interval is reduced.

Timeout_handler_interval = 5000ms

Timeout_handler_interval = 1000ms

Case 4 : Unknown Host Exception

Another scenario that was verified was giving an invalid host name, which resulted in an ‘Unknown Host Exception’, but this is handled within the 3000ms timespan.

Apart from these, there are a number of scenarios related to troubleshooting issues in endpoints. Refer for possible error scenarios in endpoints in [2]

References

[1] https://www.soapui.org/soap-mocking/getting-started.html
[2] https://docs.wso2.com/display/ESB480/Error+Handling#ErrorHandling-codes

August 8, 2016

Using SAML SSO Authorize Carbon Admin Services from Jaggery App

To give a background on the above scenario, please refer the following diagram.



A Jaggery App is deployed on WSO2 Application Server (AS) and we have enabled SAML 2.0 based SSO for the Jaggery app using WSO2 Identity Server (IS) . You can find more details on enabling SSO with IS at [1]. ESB also shares the same IDP.

What we are doing here is authenticating the Jaggery App from IDP by sending a SAML request and using the same SAML response to authenticate ESB and get a session cookie. This cookie can then be used to invoke admin services of ESB.

[JAGGERY_APP]/jagg/jaggery_acs file acts as the assertion consumer service (ACS). This is the same that is configured in the IDP. Once the request comes to jaggery_acs.jag we will validate the SAML response and get the cookie from ESB.

Once we have the SAML response, use the following code to authenticate ESB.

 //authenticate ESB
var ws = require("ws");
var requestESB = new ws.WSRequest();
var options = new Array();
options.useSOAP = 1.2;
options.action = "urn:login";
var endPoint = "https://localhost:9453/services/SAML2SSOAuthenticationService";
var payload = '<sso:login xmlns:sso="http://sso.saml2.authenticator.identity.carbon.wso2.org"><sso:authDto><xsd:response xmlns:xsd="http://dto.sso.saml2.authenticator.identity.carbon.wso2.org/xsd">' + samlResponse + '</xsd:response></sso:authDto></sso:login>';
requestESB.open(options,endPoint, false);
requestESB.send(payload);
var responseESB = requestESB.responseE4X;
var adminSession = requestESB.getResponseHeader("Set-Cookie");
session.put("esb-auth-cookie", adminSession);


Use the following code segment to invoke ESB Admin services with the above cookie

var restApiAdminUrl = site.esb.serverBaseURL + "t/" + tenantDomain + "/services/" + REST_API_ADMIN_SERVICE + "/";
var requestPayload = "<xsd:getAPIsForListing xmlns:xsd=\"http://org.apache.axis2/xsd\">"
                      +"  <xsd:pageNumber>0</xsd:pageNumber>"
                      +"  <xsd:itemsPerPage>100</xsd:itemsPerPage>"
                      +"</xsd:getAPIsForListing>";

var ws = require("ws");
var request = new ws.WSRequest();
var options = new Array();
options.useSOAP = 1.2;
options.action = "urn:getAPIsForListing";
options["HTTPHeaders"] = [{name: "cookie", value: session.get("esb-auth-cookie")}];
request.open(options, restApiAdminUrl, false);
request.send(requestPayload);
var response = request.responseE4X;



References

[1] https://docs.wso2.com/display/IS500/Configuring+Single+Sign-On+with+SAML+2.0
[2] http://wso2.com/library/articles/2016/02/article-how-to-setup-a-wso2-api-manager-store-login-with-google/

June 12, 2016

WSO2 Admin Services

WSO2 Admin Services

All WSO2 Products internally perform all of it's operations via admin services. From the management console, these admin services are called to facilitate the features.

You can find more information on this at : https://docs.wso2.com/display/Carbon420/Calling+Admin+Services+from+Apps

By default, admin services WSDLs are not exposed in a product. Therefore need to do the following to access the WSDLs.

  • Set the <HideAdminServiceWSDLs> element to false in <PRODUCT_HOME>/repository/conf/carbon.xml file and restart the server.
  • Additionally to access the OSGI console, start the server as <PRODUCT_HOME>/bin/wso2server.sh -DosgiConsole. By hitting enter on the console it will start the OSGI shell
  • osgi> listAdminServices will list all the admin services available in the product.


Invoking Admin Services

There are a few ways to invoke admin services.
  1. Invoke with SOAP UI using the WSDLs of the admin services
  2. Invoke via CURL
  3. Invoke programmatically via the service stubs.

Using SOAP UI to Invoke Admin Services

We can create a new SOAP UI project by using the WSDL. We can get the sample requests and fill up the necessary data to invoke the service operations.

Even if we are using CURL to invoke the admin services, we need to get the request format which is easiest to be done by importing the WSDL as a SOAP project to SOAP UI first.

Using CURL to Invoke Admin Services

As mentioned above, best approach is to get the request format from SOAPUI. Once you have the request format, we can invoke admin services as below.

1. First we need to authenticate. Therefore, invoke the AuthenticationAdmin service.

Request (auth.xml):

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://org.apache.synapse/xsd">
 <soapenv:Header/>
 <soapenv:Body>
      <aut:login xmlns:aut="http://authentication.services.core.carbon.wso2.org">
         <aut:username>admin</aut:username>
         <aut:password>admin</aut:password>
         <aut:remoteAddress>localhost</aut:remoteAddress>
      </aut:login>
   </soapenv:Body>
</soapenv:Envelope>

CURL command :

curl -k -v -H "SOAPAction: urn:login" -H "Content-Type: text/xml" -d @auth.xml https://localhost:9443/services/AuthenticationAdmin

Response :
*   Trying ::1...
* connect to ::1 port 9443 failed: Connection refused
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: localhost
> POST /services/AuthenticationAdmin HTTP/1.1
> Host: localhost:9443
> User-Agent: curl/7.43.0
> Accept: */*
> SOAPAction: urn:login
> Content-Type: text/xml
> Content-Length: 467
* upload completely sent off: 467 out of 467 bytes
< HTTP/1.1 200 OK
< Set-Cookie: JSESSIONID=E74FD5AB56A283CD261E26A77B9854FB; Path=/; Secure; HttpOnly
< Content-Type: text/xml;charset=UTF-8
< Transfer-Encoding: chunked
< Date: Sun, 12 Jun 2016 03:20:09 GMT
< Server: WSO2 Carbon Server
* Connection #0 to host localhost left intact

<?xml version='1.0' encoding='UTF-8'?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Body>
      <ns:loginResponse xmlns:ns="http://authentication.services.core.carbon.wso2.org">
         <ns:return>true</ns:return>
      </ns:loginResponse>
   </soapenv:Body>
</soapenv:Envelope>

At this time you should also be able to see the following command in the ESB terminal (or the relevant product's terminal)

[2016-06-12 08:52:25,931]  INFO - CarbonAuthenticationUtil 'admin@carbon.super [-1234]' logged in at [2016-06-12 08:52:25,931+0530]

2. From the response, get the JSessionID. This will be used for later requests to authenticate.

JSESSIONID=E74FD5AB56A283CD261E26A77B9854FB

3. If you are trying this with ESB, we can try invoking TemplateAdminService as below.

Request (template.xml):
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://org.apache.synapse/xsd">
   <soapenv:Header/>
   <soapenv:Body>
      <xsd:getTemplates>
         <!--Optional:-->
         <xsd:pageNumber>0</xsd:pageNumber>
         <!--Optional:-->
         <xsd:templatePerPage>200</xsd:templatePerPage>
      </xsd:getTemplates>
   </soapenv:Body>
</soapenv:Envelope>


CURL Command

curl -k -v -H "Cookie: JSESSIONID=7D8798EB0A39E18194E778461F7E3E6D" -H "SOAPAction: urn:getTemplates" -H "Content-Type: text/xml" -d @template.xml https://localhost:9443/services/TemplateAdminService

Response :
*   Trying ::1...
* connect to ::1 port 9443 failed: Connection refused
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: localhost
> POST /services/TemplateAdminService HTTP/1.1
> Host: localhost:9443
> User-Agent: curl/7.43.0
> Accept: */*
> Cookie: JSESSIONID=7D8798EB0A39E18194E778461F7E3E6D
> SOAPAction: urn:getTemplates
> Content-Type: text/xml
> Content-Length: 389
* upload completely sent off: 389 out of 389 bytes
< HTTP/1.1 200 OK
< Content-Type: text/xml;charset=UTF-8
< Transfer-Encoding: chunked
< Date: Sun, 12 Jun 2016 03:33:22 GMT
< Server: WSO2 Carbon Server
* Connection #0 to host localhost left intact

<?xml version='1.0' encoding='UTF-8'?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Body>
      <ns:getTemplatesResponse xmlns:ax2246="http://common.templates.mediation.carbon.wso2.org/xsd" xmlns:ns="http://org.apache.synapse/xsd">
         <ns:return xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ax2246:TemplateInfo">
            <ax2246:artifactContainerName xsi:nil="true"/>
            <ax2246:description xsi:nil="true"/>
            <ax2246:enableStatistics>false</ax2246:enableStatistics>
            <ax2246:enableTracing>false</ax2246:enableTracing>
            <ax2246:isEdited>false</ax2246:isEdited>
            <ax2246:name>getUserID</ax2246:name>
         </ns:return>
      </ns:getTemplatesResponse>
   </soapenv:Body>
</soapenv:Envelope>

Note : 

For all the requests, make sure to use the SOAP 1.1 which is xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" and not xmlns:soap="http://www.w3.org/2003/05/soap-envelope", since this will result in an error invoking the admin services with CURL as  : 

<?xml version='1.0' encoding='UTF-8'?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header xmlns:wsa="http://www.w3.org/2005/08/addressing">
      <wsa:Action>http://www.w3.org/2005/08/addressing/soap/fault</wsa:Action>
   </soapenv:Header>
   <soapenv:Body>
      <soapenv:Fault>
         <faultcode>soapenv:VersionMismatch</faultcode>
         <faultstring>Transport level information does not match with SOAP Message namespace URI</faultstring>
         <detail/>
      </soapenv:Fault>
   </soapenv:Body>
</soapenv:Envelope>

Using Service Stubs to Invoke Admin Services

Following is the code segment to invoke to get the templates from ESB server by using the TemplateAdminService stub. 

First step is to authenticate : 

TemplateAdminServiceStub templateAdminServiceStub = new TemplateAdminServiceStub("https://localhost:9443/services/TemplateAdminService");
ServiceClient client = templateAdminServiceStub._getServiceClient();
Options client_options = client.getOptions();
HttpTransportProperties.Authenticator authenticator = new HttpTransportProperties.Authenticator();
authenticator.setUsername("admin");
authenticator.setPassword("admin");
authenticator.setPreemptiveAuthentication(true);
client_options.setProperty(org.apache.axis2.transport.http.HTTPConstants.AUTHENTICATE, authenticator);
client.setOptions(client_options);

Next we can invoke the service operation : 

OMElement templateOMElement = templateAdminServiceStub.getTemplate();

Invoking Admin Services in Tenant Mode

To invoke admin services in tenant mode, we only need to change the admin service URL with tenant URL and the username and password to the tenant.

E.g. :
Admin Service URL :  https://localhost:9443/services/t/maheeka.com/TemplateAdminService
Username : admin@maheeka.com
Password : admin123

Troubleshooting

1. Error when invoking admin service

[2016-05-25 16:41:36,054] ERROR - Class System failure.null
java.lang.NullPointerException
at org.wso2.carbon.server.admin.module.handler.AuthorizationHandler.doAuthorization(AuthorizationHandler.java:104)
at org.wso2.carbon.server.admin.module.handler.AuthorizationHandler.invoke(AuthorizationHandler.java:87)
at org.apache.axis2.engine.Phase.invokeHandler(Phase.java:340)
at org.apache.axis2.engine.Phase.invoke(Phase.java:313)
at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:261)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:167)
at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:398)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:145)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2016-05-25 16:41:36,055] ERROR - AxisEngine System failure.
org.apache.axis2.AxisFault: System failure.
at org.wso2.carbon.server.admin.module.handler.AuthorizationHandler.invoke(AuthorizationHandler.java:93)
at org.apache.axis2.engine.Phase.invokeHandler(Phase.java:340)
at org.apache.axis2.engine.Phase.invoke(Phase.java:313)
at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:261)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:167)
at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:398)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:145)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2016-05-25 16:41:36,057] ERROR - ServerWorker Error processing POST request for : /services/CarbonAppUploader
org.apache.axis2.AxisFault: System failure.
at org.wso2.carbon.server.admin.module.handler.AuthorizationHandler.invoke(AuthorizationHandler.java:93)
at org.apache.axis2.engine.Phase.invokeHandler(Phase.java:340)
at org.apache.axis2.engine.Phase.invoke(Phase.java:313)
at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:261)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:167)
at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:398)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:145)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Solution : 
ESB or APIM has two types of transports - servlet and passthrough. The admin services are exposed through servlet transport. Therefore, we need to invoke admin services through transport port. E.g. : https://localhost:9443/services/AuthenticationAdmin. The common reason for this error is that the admin service WSDLs are exposed on passthrough port (8243).


December 17, 2015

Customize APIM Gateway Errors

Gateway is the underlying synapse engine in APIM that can take the responsibility of performing throttling, authentication, etc in order to manage the API [1]

We will consider throttle handler and auth handler for this post. The sequences for handling these reside in [APIM_HOME]/repository/deployments/synapse-configs/default/sequences/ , as _auth_failure_handler_.xml and _throttle_out_handler_.xml

When an auth error occurs or when throttle limit is exceeded, APIM directs the flow to these handlers. We can set the desired fault payload in these handlers here by using a payload factory. Consider the following sample in _throttle_out_handler_ sequence

<?xml version="1.0" encoding="UTF-8"?><sequence xmlns="http://ws.apache.org/ns/synapse" name="_throttle_out_handler_">
    <property name="error_message_type" scope="default" action="set" value="application/json"/>
    <payloadFactory media-type="xml">
      <format>
         <error xmlns="">
            <status>$1</status>
            <message>$2</message>
         </error>
      </format>
      <args>
         <arg expression="$ctx:ERROR_CODE" evaluator="xml"></arg>
         <arg expression="$ctx:ERROR_MESSAGE" evaluator="xml"></arg>
      </args>
    <sequence key="_build_"/>
    <property name="X-JWT-Assertion" scope="transport" action="remove"/>
</sequence>
Payload factory mediator is used create the custom error message. This is not mandatory if you want the default error message.  

Consider the following property which is set to get the error message in application/json. Setting messageType property is not applicable to handlers, you need to set error_message_type for this.

<property name="error_message_type" scope="default" action="set" value="application/json"/>
Reference : 

December 11, 2015

JSON Payload as String in Mediation

Due to the underlying synapse engine based on SOAP/XML message format, when a JSON payload is to be processed with ESB, it is being built as a XML message by default. In any case if we want to use the JSON payload as a string in our mediation, we will need to follow the below steps.

First the payload needs to be set as a string property as below :
<property name="JSONPayload" expression="json-eval($.)"/>
This property can then be used for any transformations as a string literal of the JSON.

Check out the sample below, which gets the response from the <call> mediator as a JSON, saves it to a string and then uses it in an xml payload. The final response of the API is XML.
<api xmlns="http://ws.apache.org/ns/synapse" name="JSONTestAPI" context="/json">
   <resource methods="GET">
      <inSequence>
         <call>
            <endpoint key="MockBackend"></endpoint>
         </call>
         <property name="JSONPayload" expression="json-eval($.)"></property>
         <log level="full">
            <property name="====JSONPayload====" expression="$ctx:JSONPayload"></property>
         </log>
         <payloadFactory media-type="xml">
            <format>
               <xmlPayload xmlns="">
                  <jsonPayload>$1</jsonPayload>
               </xmlPayload>
            </format>
            <args>
               <arg evaluator="xml" expression="$ctx:JSONPayload"></arg>
            </args>
         </payloadFactory>
         <property name="messageType" value="application/xml" scope="axis2"></property>
         <respond></respond>
      </inSequence>
   </resource>
</api>


Response from mock backend is as below :
{
  "mock":"mock service"
}

JSON payload will be logged as below when invoking the API. Notice that although the JSON payload is built to XML, the property preserves the payload as a string literal.

LogMediator To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:08b4f213-0df0-4d18-88b7-8cf3e2d1872c, Direction: request, ====JSONPayload==== = {
   "mock":"mock service"
}, Envelope: <?xml version="1.0" encoding="utf-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><jsonObject><mock>mock service</mock></jsonObject></soapenv:Body></soapenv:Envelope>

Final response from the API is as below :
<xmlPayload>
      <jsonPayload>{"mock":"mock service"}</jsonPayload>
</xmlPayload>

Additional Reference : 
http://stackoverflow.com/questions/27848342/getpayloadjson-returning-an-empty-object

November 30, 2015

Sample for Restricting Proxy with Throttle Mediator

Following is a sample of using Throttle mediator to restrict local proxy invocations. Notice that the IP has to be given as 127.0.0.1 and not "localhost".

A custom fault message can be added within onreject/makefault/reason tag (line 19-23) . Custom fault messages cannot be added for service level throttling.

<proxy name="ThrottleMediatorSample" startonload="true" trace="disable" transports="https http" xmlns="http://ws.apache.org/ns/synapse">
   <description>
   <target>
      <insequence>
         <throttle id="A">
            <policy>
               <wsp:policy wsu:id="WSO2MediatorThrottlingPolicy" xmlns:throttle="http://www.wso2.org/products/wso2commons/throttle" xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
                  <throttle:mediatorthrottleassertion>
                     <wsp:policy>
                        <throttle:id throttle:type="IP">127.0.0.1</throttle:id>
                        <wsp:policy>
                           <throttle:deny>
                        </throttle:deny></wsp:policy>
                     </wsp:policy>
                  </throttle:mediatorthrottleassertion>
               </wsp:policy>
            </policy>
            <onreject>
               <makefault response="true" version="soap11">
                  <code value="tns:Receiver" xmlns:tns="http://www.w3.org/2003/05/soap-envelope">
                     <reason value="ERROR : Restricted IP Address"/>
                  </code>
               </makefault>
               <send/>           
            </onreject>
            <onaccept>
               <send>
                  <endpoint>
                     <address uri="http://localhost:9000/services/SimpleStockQuoteService">
                     </address>
                  </endpoint>
               </send>
            </onaccept>
         </throttle>
      </insequence>
      <outsequence>
         <send>
      </send></outsequence>
   </target>
</description></proxy>

Reference :

September 13, 2015

ESB 4.9.0 - Enhanced RabbitMQ Support

WSO2 ESB 4.9.0 Released !!

Checkout the blog at : http://soatutorials.blogspot.com/2015/09/wso2-esb-490-released.html for details on the exciting new features of this release.

You can download latest ESB version from : http://wso2.com/products/enterprise-service-bus/
Product documentation is available at : https://docs.wso2.com/display/ESB490/

With this release there are a number of enhancements done in RabbitMQ transport support. Documentation for RabbitMQ transport is available at : https://docs.wso2.com/display/ESB490/RabbitMQ+AMQP+Transport

So what's new with RabbitMQ support ?

  • RabbitMQ transport is now inbuilt with WSO2 ESB. Earlier it had to be installed separately as a feature on ESB. But now it is just a matter of modifying the axis2.xml file as would for any other transport
  • Introducing RabbitMQ based message store
  • Automatic connection recovery for RabbitMQ transport incase of network failure, server shutdown etc. Just a few configurations in axis2.xml and you are good to go
  • Introducing RabbitMQ SSL transport support for sender and receiver
  • Synchronized request-response support for RabbitMQ sender. Configure a reply-to queue and RabbitMQ sender is now no longer "out-only"
  • Inbound endpoints is one of the major features of this release and we have introduced for RabbitMQ as well
  • All content-types are now supported by RabbitMQ transport

I will be writing more posts on these features in future posts.

ESB 4.9.0 - Introducing ForEach Mediator

WSO2 ESB 4.9.0 Released !!

Checkout the blog at : http://soatutorials.blogspot.com/2015/09/wso2-esb-490-released.html for details on the exciting new features of this release.

You can download latest ESB version from : http://wso2.com/products/enterprise-service-bus/
Product documentation is available at : https://docs.wso2.com/display/ESB490/

One of the new features that is released is the ForEach mediator. This post is to give an introduction to this mediator. Documentation for the ForEach mediator is available at: https://docs.wso2.com/display/ESB490/ForEach+Mediator

What happens in ForEach mediator?

  • ForEach mediator requires and xpath/jsonpath expression and a sequence (inline or referenced)
  • The original message is split to sub messages based on xpath/jsonpath expression and each such message is mediated sequentially in the mediation flow as defined by the sequence
  • ForEach mediator works in a single thread, thus a blocked execution
  • After the mediation of the sub messages are completed, they are merged back to the original message context to the original parent

ForEach or Iterate Mediator?

  • ForEach mediator and Iterate mediator may sound similar at first. But there are significant differences
  • Iterate mediator will always have to be accompanied by an Aggregate mediator. But ForEach mediator will complete all the processing within the mediator and will have the full message context after the mediation
  • Iterate mediator will allow sending a message to an endpoint where as ForEach mediator will not allow Call, Send or CallOut mediators. ForEach mediator is mostly useful for payload transformation use cases.
  • Xpath/jsonpath expressions can be used to conditionally select elements to be iterated in both mediators
  • ForEach mediator will not split the message flow unlike Iterate mediator. ForEach will execute all iterations in a single thread
  • ForEach supports modifying the original payload. Iterate will be used for situations where split messages will be sent to a target and collected by aggregate in a different flow. ForEach is for modifying a split message. ForEach also eliminates the need for complex XSLT mediators. 

Check out ESB documentation for samples on using ForEach mediator.

August 10, 2015

Using Script Mediator to Modify a JSON Payload

Environment :  ESB 4.8.0
Assuming the following properties are defined :
Property1 : {"object1": "this is object1"}
Property2 : {"sample": [123, "ABC", "456", "A12"]}

Following is the way to use script mediator to append a json array to an existing payload.
<script language="js">
var prop1 = eval('('+mc.getProperty("Property1") +')');
var payload ={};
var prop2 =eval('('+mc.getProperty("Property2")+ ')');
prop1["object2"]=prop2;
payload["final"]=prop1;
mc.setPayloadJSON(payload);
</script>
Following is the final payload.
{"final":{"object1":"this is object1","object2":{"sample":[123,"ABC","456","A12"]}}}

July 26, 2015

WS-Addressing with MessageID using SOAPUI and ESB

For all requests received by WSO2 ESB, a new message context is created with a new Message ID. This message ID is useful if you need to use it as a unique identification or for a correlation identification at some point.

However, if the request uses WS-Addressing, ESB will reuse the same Message ID provided with the request. 

You can send a request with WS-Addressing as in the following in example. Enter a MessageID and use a log in the relevant proxy in ESB to view the MessageID using get-property('MessageID')

If there is any requirement to get a new MessageID, despite using WS-Addressing, then you will need to include your complete message flow within a clone mediator, which will create a new message context and thus a new MessageID.

Example :

Following file will send the request to SimpleStockQuoteService (sample backend available with ESB) and write the response to a file. Here the file name is taken from MessageID. If the MessageID was not unique it will be overwriting the same file. Please note that this is assuming MessageID does not have any importance when serving the message.

Sending the request using SOAPUI :
  • Enable WS-A addressing
  • Enable Add default wsa:action
  • Enable Add default wsa:To
  • Enter MessageID in urn:XXXXXXXX or urn:uuid:XXXXXXXX (urn:uuid is the format used in WSO2 ESB)


<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="StockProxy"
       transports="http,https"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <property name="messageid"
                   expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.xml')"/>
         <send>
            <endpoint>
               <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
            </endpoint>
         </send>
      </inSequence>
      <outSequence>
         <property name="OUT_ONLY" value="true"/>
         <property name="transport.vfs.ReplyFileName"
                   expression="get-property('messageid')"
                   scope="transport"/>
         <property name="transport.vfs.ContentType" value="text/xml" scope="transport"/>
         <property name="ClientApiNonBlocking" scope="axis2" action="remove"/>
         <send>
            <endpoint>
               <address uri="vfs:file:///Users/maheeka/test/out"/>
            </endpoint>
         </send>
         <send/>
      </outSequence>
   </target>
   <description/>
</proxy>
       
Solution : 
To generate a new MessageID, we are using a clone mediator in between.

<proxy name="StockProxy" startOnLoad="true" statistics="disable" trace="disable" transports="http,https" xmlns="http://ws.apache.org/ns/synapse">
  <target>
    <inSequence>
      <property expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.xml')" name="messageid"/>
      <send>
        <endpoint>
          <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
        </endpoint>
      </send>
    </inSequence>
    <outSequence>
      <clone>
        <target>
          <sequence>
            <property name="OUT_ONLY" value="true"/>
            <property expression="get-property('messageid')" name="transport.vfs.ReplyFileName" scope="transport"/>
            <property name="transport.vfs.ContentType" scope="transport" value="text/xml"/>
            <property action="remove" name="ClientApiNonBlocking" scope="axis2"/>
            <send>
              <endpoint>
                <address uri="vfs:file:///Users/maheeka/test/out"/>
              </endpoint>
            </send>
            <send/>
          </sequence>
        </target>
      </clone>
    </outSequence>
  </target>
  <description/>
</proxy>

ClientApiNonBlocking property needs to be removed so that the primary thread will do the send to the VFS endpoint. For more details on this property refer : https://docs.wso2.com/display/ESB481/Generic+Properties

Reference :

July 25, 2015

Optional Query Parameters in APIM and ESB APIs

When adding an API resource, url-template or url-mapping specifies the url pattern for to accept requests. In case of optional parameters, this needs to be handled too in the url pattern. However, this is not directly possible in APIs for both WSO2 ESB and APIM with a single resource.

Therefore, need to define two separate resources, one to accept additional(optional) parameters and the other to accept mandatory parameters. In /access_tokens/{entityRef}* , * denotes the parameters that may follow.

<api xmlns="http://ws.apache.org/ns/synapse" name="UserAPI" context="/users">
   <resource methods="GET" uri-template="/user/{userid}*">
      <inSequence>
         <property name="userid" expression="$ctx:uri.var.userid" scope="default" type="STRING"></property>
         <property name="name" expression="$ctx:query.param.name"></property>
         <log>
            <property name="STATUS" value="Request received for /users/{userid}*"></property>
         </log>
         <drop></drop>
      </inSequence>
   </resource>
   <resource methods="GET" uri-template="/user/{userid}">
      <inSequence>
         <property name="userid" expression="$ctx:uri.var.userid" scope="default" type="STRING"></property>
         <log>
            <property name="STATUS" value="Request received for /user/{userid}"></property>
         </log>
      </inSequence>
   </resource>
</api>

The following type of requests can be served with above API definition.

July 13, 2015

WSO2 ESB with SAP in OSX

Configure WSO2 ESB for SAP

Refer below steps while following the official documentation [1] to do the configurations in OSX.

1. Download SAP JCO for OSX from http://service.sap.com/connectors

2. Copy the sapjco3.jar and sapidoc3.jar files to [ESB_HOME]/repository/components/lib folder

3. Setup classpath to SAP as follows : 
export LD_LIBRARY_PATH=/Users/maheeka/sapjco3/
export CLASSPATH=/Users/maheeka/sapjco3/sapjco3.jar
For additional details on configuring classpath refer [2]

4. Enable SAP transport receiver/sender and create *.dest and *.server files as mentioned in documentation [1] (will be referred to as SAP.dest and SAP.server below)

5. Start ESB with the following command
sh [ESB_HOME]bin/wso2server.sh -Djava.library.path=<path_sapjco3>
Give path to sapjco3 folder downloaded in step 1.

[1] https://docs.wso2.com/display/ESB481/SAP+Integration
[2] http://www.mobility-platform.com/en/documentation/administration/system-landscape-connector/sap-connector/sap-jco-installation/

Configure SAP with SAPGUI

SAPGUI is the client used for configuring SAP.

1. Download SAP GUI for OSX : PlatinGUI740_0-20012037.JAR and  (documentation) files from : http://scn.sap.com/community/gui/blog/2013/05/29/sap-gui-730-download
(SAP GUI 7.3.0 requires jdk 7 and SAP GUI 7.4.0 requires jdk 8 minimum)

2. Issue command : java -jar PlatinGUI740_0-20012037.JAR install
(Download PlatinManual_0-20008876.ZIP from the above location for the manual containing installation instructions)

3. Go through the installation wizard to install SAP GUI

4. Configure the following in SAP to connect to ESB via SAP GUI. First configure a connection and do the following steps.

  1. Create a TCP/IP connection and test the connection (RFC Destinations > TCP/IP Connections)
  2. Create a port (Port definition)
  3. Create a logical system (Maintaining Logical Systems)
  4. Create a partner agreement for the logical system (Partner Profiles)
  5. Create outbound and inbound parameters for partner agreement
You can also use transaction codes instead of using the user menu for this step. Transaction codes are available at : http://wiki.scn.sap.com/wiki/display/ABAP/List+of+Transaction+codes?original_fqdn=wiki.sdn.sap.com

Sample IDoc Sender

Use the following proxy as a sample IDOC sender.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="IDOCProxy"
       transports="http"
       startOnLoad="true"
       trace="disable">
   <description/>
   <target>
      <inSequence>
         <xslt key="idoc.xslt"/>
         <property name="FORCE_SC_ACCEPT" value="true"/>
         <property name="OUT_ONLY" value="true"/>
         <send>
            <endpoint>
               <address uri="idoc:/SAP"/>
            </endpoint>
         </send>
      </inSequence>
   </target>
</proxy>


"idoc.xslt" will create a IDOC payload and the proxy will send the request to SAP.

Invoke the proxy with : curl -v -X POST "http://localhost:8280/services/IDOCProxy"
If the invoke is successful, you can view the response in IDOC List menu.


Troubleshooting

1. Enable tracing for SAP by setting the trace property in *.dest file as : jco.client.trace=1
2. When step (3) and (5) in "Configure SAP with WSO2 ESB" is not done correctly : 
FATAL - CarbonServerManager WSO2 Carbon initialization Failed
java.lang.ExceptionInInitializerError: JCo initialization failed with java.lang.UnsatisfiedLinkError: no sapjco3 in java.library.path
   at com.sap.conn.jco.rt.MiddlewareJavaRfc.<clinit>(MiddlewareJavaRfc.java:229)
   at com.sap.conn.jco.rt.DefaultJCoRuntime.initialize(DefaultJCoRuntime.java:98)
   at com.sap.conn.jco.rt.JCoRuntimeFactory.<clinit>(JCoRuntimeFactory.java:23)
   at java.lang.Class.forName0(Native Method)
   at java.lang.Class.forName(Class.java:171)
   at com.sap.conn.jco.JCo.createJCo(JCo.java:52)
   at com.sap.conn.jco.JCo.<clinit>(JCo.java:26)
   at java.lang.Class.forName0(Native Method)
   at java.lang.Class.forName(Class.java:171)
   at com.sap.conn.idoc.jco.JCoIDoc.<clinit>(JCoIDoc.java:140)
   at org.wso2.carbon.transports.sap.idoc.DefaultIDocXMLMapper.<init>(DefaultIDocXMLMapper.java:49)
   at org.wso2.carbon.transports.sap.SAPTransportSender.<init>(SAPTransportSender.java:62)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at java.lang.Class.newInstance0(Class.java:357)
   at java.lang.Class.newInstance(Class.java:310)
   at org.apache.axis2.deployment.AxisConfigBuilder.processTransportSenders(AxisConfigBuilder.java:688)
   at org.apache.axis2.deployment.AxisConfigBuilder.populateConfig(AxisConfigBuilder.java:124)
   at org.wso2.carbon.core.CarbonAxisConfigurator.populateAxisConfiguration(CarbonAxisConfigurator.java:308)
   at org.wso2.carbon.core.CarbonAxisConfigurator.getAxisConfiguration(CarbonAxisConfigurator.java:188)
   at org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContext(ConfigurationContextFactory.java:64)
   at org.wso2.carbon.core.CarbonConfigurationContextFactory.createNewConfigurationContext(CarbonConfigurationContextFactory.java:65)
   at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:398)
   at org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:219)
   at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:77)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
   at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
   at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:347)
   at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
   at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
   at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
   at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
   at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
   at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
   at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
   at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
   at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
   at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
   at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
   at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
   at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
   at org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81)
   at org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60)
   at org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40)
   at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38)
   at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1267)
   at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1186)
   at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1081)
   at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5027)
   at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
   at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

April 10, 2015

Git Useful Commands - 3

Create a branch from an old commit
git branch <branch_name> <sha1_of_commit>

or by using symbolic ref.

git branch <branch_name> HEAD~3

Push a local branch to repository

$git push origin <localbranch>

Set up tracking with the remote branch :

Tracking is useful to do pull, fetch, etc. 

git branch --set-upstream-to=origin/<branch> <localbranch>


Reset branch with remote branch


git reset --hard upstream/branch

This will reset the current branch with the given upstream branch. Then, push the branch.

git push -f origin branch

Create a new branch

Checkout the branch you need to branch from

git checkout -b branch1
git push origin branch1

Git Remove Untracked Files
git clean -df

Git Untrack Files
git rm -r --cached <filename>

Checkout from Upstream Tag
To create a branch from an upstream tag, first fetch from upstream with git fetch upstream
Now checkout the tag to a new branch

git checkout <tagname> && git checkout -b <branchname>

March 6, 2015

Git - Useful Commands - 2

Git Verify a Pull Request Locally

Before merging a pull request to your repository, it is best to verify the pull request locally. Therefore, do the following :

$git fetch origin pull/X/head:pullX

Replace X with the pull request number. Here we assume that you are directly working on the repository the pull request was sent to. Now a new branch is created in your local repository as pullX and it contains the changes of the pull request. Now you need to merge this with the current branch or any other branch to verify. Assuming you want to merge to master branch,

$git checkout master

$git merge pullX

Now you can verify the pull request and decide on the next step for committing.

Git Verify a Pull Request of Non Default Repository Locally

Assume you are on repository A. The pull request has been sent to repository A2. A2 is a fork of A. 
Now, you will have to do the following :

$git remote add upstream [url to A2]

This will add A2 as a remote branch referred by upstream.

Now you can fetch the pull request same like before using :

$git fetch upstream pull/X/head:pullX

Notice that, now you are fetching the pull request from the upstream repository. Rest of the process is same as above. 

Git Rollback a Pushed Commit
$git reset --hard <old-commit-id>
$git push -f origin <branch>

Git Rollback Last Pushed Commit
$git reset HEAD^ --hard
$git push -f origin <branch>

Git Remove a File from Last Commit
If you mistakenly committed a file and you want to remove it from your last commit, do the following : 

Checkout the required commit version of the particular file. If it is a new file, it is okay to remove the file with "rm"
$git checkout <commit_id> <path_to_file>

Now amend the commit
$git commit --amend

If you have already pushed your commit,
$git push -f origin <branch>

Amend Last Commit
--amend option can be used to modify last commit in anyway. Example : modify a file, add a missed file, change commit message, etc.

If you do not want to change the commit message,
$git commit --amend --no-edit