December 17, 2015

Customize APIM Gateway Errors

Gateway is the underlying synapse engine in APIM that can take the responsibility of performing throttling, authentication, etc in order to manage the API [1]

We will consider throttle handler and auth handler for this post. The sequences for handling these reside in [APIM_HOME]/repository/deployments/synapse-configs/default/sequences/ , as _auth_failure_handler_.xml and _throttle_out_handler_.xml

When an auth error occurs or when throttle limit is exceeded, APIM directs the flow to these handlers. We can set the desired fault payload in these handlers here by using a payload factory. Consider the following sample in _throttle_out_handler_ sequence

<?xml version="1.0" encoding="UTF-8"?><sequence xmlns="http://ws.apache.org/ns/synapse" name="_throttle_out_handler_">
    <property name="error_message_type" scope="default" action="set" value="application/json"/>
    <payloadFactory media-type="xml">
      <format>
         <error xmlns="">
            <status>$1</status>
            <message>$2</message>
         </error>
      </format>
      <args>
         <arg expression="$ctx:ERROR_CODE" evaluator="xml"></arg>
         <arg expression="$ctx:ERROR_MESSAGE" evaluator="xml"></arg>
      </args>
    <sequence key="_build_"/>
    <property name="X-JWT-Assertion" scope="transport" action="remove"/>
</sequence>
Payload factory mediator is used create the custom error message. This is not mandatory if you want the default error message.  

Consider the following property which is set to get the error message in application/json. Setting messageType property is not applicable to handlers, you need to set error_message_type for this.

<property name="error_message_type" scope="default" action="set" value="application/json"/>
Reference : 

December 11, 2015

JSON Payload as String in Mediation

Due to the underlying synapse engine based on SOAP/XML message format, when a JSON payload is to be processed with ESB, it is being built as a XML message by default. In any case if we want to use the JSON payload as a string in our mediation, we will need to follow the below steps.

First the payload needs to be set as a string property as below :
<property name="JSONPayload" expression="json-eval($.)"/>
This property can then be used for any transformations as a string literal of the JSON.

Check out the sample below, which gets the response from the <call> mediator as a JSON, saves it to a string and then uses it in an xml payload. The final response of the API is XML.
<api xmlns="http://ws.apache.org/ns/synapse" name="JSONTestAPI" context="/json">
   <resource methods="GET">
      <inSequence>
         <call>
            <endpoint key="MockBackend"></endpoint>
         </call>
         <property name="JSONPayload" expression="json-eval($.)"></property>
         <log level="full">
            <property name="====JSONPayload====" expression="$ctx:JSONPayload"></property>
         </log>
         <payloadFactory media-type="xml">
            <format>
               <xmlPayload xmlns="">
                  <jsonPayload>$1</jsonPayload>
               </xmlPayload>
            </format>
            <args>
               <arg evaluator="xml" expression="$ctx:JSONPayload"></arg>
            </args>
         </payloadFactory>
         <property name="messageType" value="application/xml" scope="axis2"></property>
         <respond></respond>
      </inSequence>
   </resource>
</api>


Response from mock backend is as below :
{
  "mock":"mock service"
}

JSON payload will be logged as below when invoking the API. Notice that although the JSON payload is built to XML, the property preserves the payload as a string literal.

LogMediator To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:08b4f213-0df0-4d18-88b7-8cf3e2d1872c, Direction: request, ====JSONPayload==== = {
   "mock":"mock service"
}, Envelope: <?xml version="1.0" encoding="utf-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><jsonObject><mock>mock service</mock></jsonObject></soapenv:Body></soapenv:Envelope>

Final response from the API is as below :
<xmlPayload>
      <jsonPayload>{"mock":"mock service"}</jsonPayload>
</xmlPayload>

Additional Reference : 
http://stackoverflow.com/questions/27848342/getpayloadjson-returning-an-empty-object

November 30, 2015

Sample for Restricting Proxy with Throttle Mediator

Following is a sample of using Throttle mediator to restrict local proxy invocations. Notice that the IP has to be given as 127.0.0.1 and not "localhost".

A custom fault message can be added within onreject/makefault/reason tag (line 19-23) . Custom fault messages cannot be added for service level throttling.

<proxy name="ThrottleMediatorSample" startonload="true" trace="disable" transports="https http" xmlns="http://ws.apache.org/ns/synapse">
   <description>
   <target>
      <insequence>
         <throttle id="A">
            <policy>
               <wsp:policy wsu:id="WSO2MediatorThrottlingPolicy" xmlns:throttle="http://www.wso2.org/products/wso2commons/throttle" xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
                  <throttle:mediatorthrottleassertion>
                     <wsp:policy>
                        <throttle:id throttle:type="IP">127.0.0.1</throttle:id>
                        <wsp:policy>
                           <throttle:deny>
                        </throttle:deny></wsp:policy>
                     </wsp:policy>
                  </throttle:mediatorthrottleassertion>
               </wsp:policy>
            </policy>
            <onreject>
               <makefault response="true" version="soap11">
                  <code value="tns:Receiver" xmlns:tns="http://www.w3.org/2003/05/soap-envelope">
                     <reason value="ERROR : Restricted IP Address"/>
                  </code>
               </makefault>
               <send/>           
            </onreject>
            <onaccept>
               <send>
                  <endpoint>
                     <address uri="http://localhost:9000/services/SimpleStockQuoteService">
                     </address>
                  </endpoint>
               </send>
            </onaccept>
         </throttle>
      </insequence>
      <outsequence>
         <send>
      </send></outsequence>
   </target>
</description></proxy>

Reference :

September 13, 2015

ESB 4.9.0 - Enhanced RabbitMQ Support

WSO2 ESB 4.9.0 Released !!

Checkout the blog at : http://soatutorials.blogspot.com/2015/09/wso2-esb-490-released.html for details on the exciting new features of this release.

You can download latest ESB version from : http://wso2.com/products/enterprise-service-bus/
Product documentation is available at : https://docs.wso2.com/display/ESB490/

With this release there are a number of enhancements done in RabbitMQ transport support. Documentation for RabbitMQ transport is available at : https://docs.wso2.com/display/ESB490/RabbitMQ+AMQP+Transport

So what's new with RabbitMQ support ?

  • RabbitMQ transport is now inbuilt with WSO2 ESB. Earlier it had to be installed separately as a feature on ESB. But now it is just a matter of modifying the axis2.xml file as would for any other transport
  • Introducing RabbitMQ based message store
  • Automatic connection recovery for RabbitMQ transport incase of network failure, server shutdown etc. Just a few configurations in axis2.xml and you are good to go
  • Introducing RabbitMQ SSL transport support for sender and receiver
  • Synchronized request-response support for RabbitMQ sender. Configure a reply-to queue and RabbitMQ sender is now no longer "out-only"
  • Inbound endpoints is one of the major features of this release and we have introduced for RabbitMQ as well
  • All content-types are now supported by RabbitMQ transport

I will be writing more posts on these features in future posts.

ESB 4.9.0 - Introducing ForEach Mediator

WSO2 ESB 4.9.0 Released !!

Checkout the blog at : http://soatutorials.blogspot.com/2015/09/wso2-esb-490-released.html for details on the exciting new features of this release.

You can download latest ESB version from : http://wso2.com/products/enterprise-service-bus/
Product documentation is available at : https://docs.wso2.com/display/ESB490/

One of the new features that is released is the ForEach mediator. This post is to give an introduction to this mediator. Documentation for the ForEach mediator is available at: https://docs.wso2.com/display/ESB490/ForEach+Mediator

What happens in ForEach mediator?

  • ForEach mediator requires and xpath/jsonpath expression and a sequence (inline or referenced)
  • The original message is split to sub messages based on xpath/jsonpath expression and each such message is mediated sequentially in the mediation flow as defined by the sequence
  • ForEach mediator works in a single thread, thus a blocked execution
  • After the mediation of the sub messages are completed, they are merged back to the original message context to the original parent

ForEach or Iterate Mediator?

  • ForEach mediator and Iterate mediator may sound similar at first. But there are significant differences
  • Iterate mediator will always have to be accompanied by an Aggregate mediator. But ForEach mediator will complete all the processing within the mediator and will have the full message context after the mediation
  • Iterate mediator will allow sending a message to an endpoint where as ForEach mediator will not allow Call, Send or CallOut mediators. ForEach mediator is mostly useful for payload transformation use cases.
  • Xpath/jsonpath expressions can be used to conditionally select elements to be iterated in both mediators
  • ForEach mediator will not split the message flow unlike Iterate mediator. ForEach will execute all iterations in a single thread
  • ForEach supports modifying the original payload. Iterate will be used for situations where split messages will be sent to a target and collected by aggregate in a different flow. ForEach is for modifying a split message. ForEach also eliminates the need for complex XSLT mediators. 

Check out ESB documentation for samples on using ForEach mediator.

August 10, 2015

Using Script Mediator to Modify a JSON Payload

Environment :  ESB 4.8.0
Assuming the following properties are defined :
Property1 : {"object1": "this is object1"}
Property2 : {"sample": [123, "ABC", "456", "A12"]}

Following is the way to use script mediator to append a json array to an existing payload.
<script language="js">
var prop1 = eval('('+mc.getProperty("Property1") +')');
var payload ={};
var prop2 =eval('('+mc.getProperty("Property2")+ ')');
prop1["object2"]=prop2;
payload["final"]=prop1;
mc.setPayloadJSON(payload);
</script>
Following is the final payload.
{"final":{"object1":"this is object1","object2":{"sample":[123,"ABC","456","A12"]}}}

July 26, 2015

WS-Addressing with MessageID using SOAPUI and ESB

For all requests received by WSO2 ESB, a new message context is created with a new Message ID. This message ID is useful if you need to use it as a unique identification or for a correlation identification at some point.

However, if the request uses WS-Addressing, ESB will reuse the same Message ID provided with the request. 

You can send a request with WS-Addressing as in the following in example. Enter a MessageID and use a log in the relevant proxy in ESB to view the MessageID using get-property('MessageID')

If there is any requirement to get a new MessageID, despite using WS-Addressing, then you will need to include your complete message flow within a clone mediator, which will create a new message context and thus a new MessageID.

Example :

Following file will send the request to SimpleStockQuoteService (sample backend available with ESB) and write the response to a file. Here the file name is taken from MessageID. If the MessageID was not unique it will be overwriting the same file. Please note that this is assuming MessageID does not have any importance when serving the message.

Sending the request using SOAPUI :
  • Enable WS-A addressing
  • Enable Add default wsa:action
  • Enable Add default wsa:To
  • Enter MessageID in urn:XXXXXXXX or urn:uuid:XXXXXXXX (urn:uuid is the format used in WSO2 ESB)


<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="StockProxy"
       transports="http,https"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <property name="messageid"
                   expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.xml')"/>
         <send>
            <endpoint>
               <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
            </endpoint>
         </send>
      </inSequence>
      <outSequence>
         <property name="OUT_ONLY" value="true"/>
         <property name="transport.vfs.ReplyFileName"
                   expression="get-property('messageid')"
                   scope="transport"/>
         <property name="transport.vfs.ContentType" value="text/xml" scope="transport"/>
         <property name="ClientApiNonBlocking" scope="axis2" action="remove"/>
         <send>
            <endpoint>
               <address uri="vfs:file:///Users/maheeka/test/out"/>
            </endpoint>
         </send>
         <send/>
      </outSequence>
   </target>
   <description/>
</proxy>
       
Solution : 
To generate a new MessageID, we are using a clone mediator in between.

<proxy name="StockProxy" startOnLoad="true" statistics="disable" trace="disable" transports="http,https" xmlns="http://ws.apache.org/ns/synapse">
  <target>
    <inSequence>
      <property expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.xml')" name="messageid"/>
      <send>
        <endpoint>
          <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
        </endpoint>
      </send>
    </inSequence>
    <outSequence>
      <clone>
        <target>
          <sequence>
            <property name="OUT_ONLY" value="true"/>
            <property expression="get-property('messageid')" name="transport.vfs.ReplyFileName" scope="transport"/>
            <property name="transport.vfs.ContentType" scope="transport" value="text/xml"/>
            <property action="remove" name="ClientApiNonBlocking" scope="axis2"/>
            <send>
              <endpoint>
                <address uri="vfs:file:///Users/maheeka/test/out"/>
              </endpoint>
            </send>
            <send/>
          </sequence>
        </target>
      </clone>
    </outSequence>
  </target>
  <description/>
</proxy>

ClientApiNonBlocking property needs to be removed so that the primary thread will do the send to the VFS endpoint. For more details on this property refer : https://docs.wso2.com/display/ESB481/Generic+Properties

Reference :

July 25, 2015

Optional Query Parameters in APIM and ESB APIs

When adding an API resource, url-template or url-mapping specifies the url pattern for to accept requests. In case of optional parameters, this needs to be handled too in the url pattern. However, this is not directly possible in APIs for both WSO2 ESB and APIM with a single resource.

Therefore, need to define two separate resources, one to accept additional(optional) parameters and the other to accept mandatory parameters. In /access_tokens/{entityRef}* , * denotes the parameters that may follow.

<api xmlns="http://ws.apache.org/ns/synapse" name="UserAPI" context="/users">
   <resource methods="GET" uri-template="/user/{userid}*">
      <inSequence>
         <property name="userid" expression="$ctx:uri.var.userid" scope="default" type="STRING"></property>
         <property name="name" expression="$ctx:query.param.name"></property>
         <log>
            <property name="STATUS" value="Request received for /users/{userid}*"></property>
         </log>
         <drop></drop>
      </inSequence>
   </resource>
   <resource methods="GET" uri-template="/user/{userid}">
      <inSequence>
         <property name="userid" expression="$ctx:uri.var.userid" scope="default" type="STRING"></property>
         <log>
            <property name="STATUS" value="Request received for /user/{userid}"></property>
         </log>
      </inSequence>
   </resource>
</api>

The following type of requests can be served with above API definition.

July 13, 2015

WSO2 ESB with SAP in OSX

Configure WSO2 ESB for SAP

Refer below steps while following the official documentation [1] to do the configurations in OSX.

1. Download SAP JCO for OSX from http://service.sap.com/connectors

2. Copy the sapjco3.jar and sapidoc3.jar files to [ESB_HOME]/repository/components/lib folder

3. Setup classpath to SAP as follows : 
export LD_LIBRARY_PATH=/Users/maheeka/sapjco3/
export CLASSPATH=/Users/maheeka/sapjco3/sapjco3.jar
For additional details on configuring classpath refer [2]

4. Enable SAP transport receiver/sender and create *.dest and *.server files as mentioned in documentation [1] (will be referred to as SAP.dest and SAP.server below)

5. Start ESB with the following command
sh [ESB_HOME]bin/wso2server.sh -Djava.library.path=<path_sapjco3>
Give path to sapjco3 folder downloaded in step 1.

[1] https://docs.wso2.com/display/ESB481/SAP+Integration
[2] http://www.mobility-platform.com/en/documentation/administration/system-landscape-connector/sap-connector/sap-jco-installation/

Configure SAP with SAPGUI

SAPGUI is the client used for configuring SAP.

1. Download SAP GUI for OSX : PlatinGUI740_0-20012037.JAR and  (documentation) files from : http://scn.sap.com/community/gui/blog/2013/05/29/sap-gui-730-download
(SAP GUI 7.3.0 requires jdk 7 and SAP GUI 7.4.0 requires jdk 8 minimum)

2. Issue command : java -jar PlatinGUI740_0-20012037.JAR install
(Download PlatinManual_0-20008876.ZIP from the above location for the manual containing installation instructions)

3. Go through the installation wizard to install SAP GUI

4. Configure the following in SAP to connect to ESB via SAP GUI. First configure a connection and do the following steps.

  1. Create a TCP/IP connection and test the connection (RFC Destinations > TCP/IP Connections)
  2. Create a port (Port definition)
  3. Create a logical system (Maintaining Logical Systems)
  4. Create a partner agreement for the logical system (Partner Profiles)
  5. Create outbound and inbound parameters for partner agreement
You can also use transaction codes instead of using the user menu for this step. Transaction codes are available at : http://wiki.scn.sap.com/wiki/display/ABAP/List+of+Transaction+codes?original_fqdn=wiki.sdn.sap.com

Sample IDoc Sender

Use the following proxy as a sample IDOC sender.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="IDOCProxy"
       transports="http"
       startOnLoad="true"
       trace="disable">
   <description/>
   <target>
      <inSequence>
         <xslt key="idoc.xslt"/>
         <property name="FORCE_SC_ACCEPT" value="true"/>
         <property name="OUT_ONLY" value="true"/>
         <send>
            <endpoint>
               <address uri="idoc:/SAP"/>
            </endpoint>
         </send>
      </inSequence>
   </target>
</proxy>


"idoc.xslt" will create a IDOC payload and the proxy will send the request to SAP.

Invoke the proxy with : curl -v -X POST "http://localhost:8280/services/IDOCProxy"
If the invoke is successful, you can view the response in IDOC List menu.


Troubleshooting

1. Enable tracing for SAP by setting the trace property in *.dest file as : jco.client.trace=1
2. When step (3) and (5) in "Configure SAP with WSO2 ESB" is not done correctly : 
FATAL - CarbonServerManager WSO2 Carbon initialization Failed
java.lang.ExceptionInInitializerError: JCo initialization failed with java.lang.UnsatisfiedLinkError: no sapjco3 in java.library.path
   at com.sap.conn.jco.rt.MiddlewareJavaRfc.<clinit>(MiddlewareJavaRfc.java:229)
   at com.sap.conn.jco.rt.DefaultJCoRuntime.initialize(DefaultJCoRuntime.java:98)
   at com.sap.conn.jco.rt.JCoRuntimeFactory.<clinit>(JCoRuntimeFactory.java:23)
   at java.lang.Class.forName0(Native Method)
   at java.lang.Class.forName(Class.java:171)
   at com.sap.conn.jco.JCo.createJCo(JCo.java:52)
   at com.sap.conn.jco.JCo.<clinit>(JCo.java:26)
   at java.lang.Class.forName0(Native Method)
   at java.lang.Class.forName(Class.java:171)
   at com.sap.conn.idoc.jco.JCoIDoc.<clinit>(JCoIDoc.java:140)
   at org.wso2.carbon.transports.sap.idoc.DefaultIDocXMLMapper.<init>(DefaultIDocXMLMapper.java:49)
   at org.wso2.carbon.transports.sap.SAPTransportSender.<init>(SAPTransportSender.java:62)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at java.lang.Class.newInstance0(Class.java:357)
   at java.lang.Class.newInstance(Class.java:310)
   at org.apache.axis2.deployment.AxisConfigBuilder.processTransportSenders(AxisConfigBuilder.java:688)
   at org.apache.axis2.deployment.AxisConfigBuilder.populateConfig(AxisConfigBuilder.java:124)
   at org.wso2.carbon.core.CarbonAxisConfigurator.populateAxisConfiguration(CarbonAxisConfigurator.java:308)
   at org.wso2.carbon.core.CarbonAxisConfigurator.getAxisConfiguration(CarbonAxisConfigurator.java:188)
   at org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContext(ConfigurationContextFactory.java:64)
   at org.wso2.carbon.core.CarbonConfigurationContextFactory.createNewConfigurationContext(CarbonConfigurationContextFactory.java:65)
   at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:398)
   at org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:219)
   at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:77)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
   at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
   at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:347)
   at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
   at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
   at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
   at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
   at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
   at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
   at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
   at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
   at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
   at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
   at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
   at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
   at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
   at org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81)
   at org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60)
   at org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40)
   at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38)
   at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1267)
   at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1186)
   at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1081)
   at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5027)
   at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
   at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

April 10, 2015

Git Useful Commands - 3

Create a branch from an old commit
git branch <branch_name> <sha1_of_commit>

or by using symbolic ref.

git branch <branch_name> HEAD~3

Push a local branch to repository

$git push origin <localbranch>

Set up tracking with the remote branch :

Tracking is useful to do pull, fetch, etc. 

git branch --set-upstream-to=origin/<branch> <localbranch>


Reset branch with remote branch


git reset --hard upstream/branch

This will reset the current branch with the given upstream branch. Then, push the branch.

git push -f origin branch

Create a new branch

Checkout the branch you need to branch from

git checkout -b branch1
git push origin branch1

Git Remove Untracked Files
git clean -df

Git Untrack Files
git rm -r --cached <filename>

Checkout from Upstream Tag
To create a branch from an upstream tag, first fetch from upstream with git fetch upstream
Now checkout the tag to a new branch

git checkout <tagname> && git checkout -b <branchname>

March 6, 2015

Git - Useful Commands - 2

Git Verify a Pull Request Locally

Before merging a pull request to your repository, it is best to verify the pull request locally. Therefore, do the following :

$git fetch origin pull/X/head:pullX

Replace X with the pull request number. Here we assume that you are directly working on the repository the pull request was sent to. Now a new branch is created in your local repository as pullX and it contains the changes of the pull request. Now you need to merge this with the current branch or any other branch to verify. Assuming you want to merge to master branch,

$git checkout master

$git merge pullX

Now you can verify the pull request and decide on the next step for committing.

Git Verify a Pull Request of Non Default Repository Locally

Assume you are on repository A. The pull request has been sent to repository A2. A2 is a fork of A. 
Now, you will have to do the following :

$git remote add upstream [url to A2]

This will add A2 as a remote branch referred by upstream.

Now you can fetch the pull request same like before using :

$git fetch upstream pull/X/head:pullX

Notice that, now you are fetching the pull request from the upstream repository. Rest of the process is same as above. 

Git Rollback a Pushed Commit
$git reset --hard <old-commit-id>
$git push -f origin <branch>

Git Rollback Last Pushed Commit
$git reset HEAD^ --hard
$git push -f origin <branch>

Git Remove a File from Last Commit
If you mistakenly committed a file and you want to remove it from your last commit, do the following : 

Checkout the required commit version of the particular file. If it is a new file, it is okay to remove the file with "rm"
$git checkout <commit_id> <path_to_file>

Now amend the commit
$git commit --amend

If you have already pushed your commit,
$git push -f origin <branch>

Amend Last Commit
--amend option can be used to modify last commit in anyway. Example : modify a file, add a missed file, change commit message, etc.

If you do not want to change the commit message,
$git commit --amend --no-edit

March 2, 2015

WSO2 ESB - Running Integration Tests

Integration tests for WSO2 ESB is available in product-esb repository at https://github.com/wso2/product-esb.

You can clone this module and build ESB product with mvn clean install. The product will be created in [PRODUCT_ESB]/modules/distribution/target folder as wso2esb-[version].zip file.

This will also run all the integration tests for the product and you can find the surefire reports for the product at [PRODUCT_ESB]/modules/integration/tests-integration/[relevant_module_folder]/target/surefire-reports folder.

If you are executing tests by module, you can navigate to the relevant module and issue mvn clean install. However, before doing so, issue a mvn clean install -Dmaven.test.skip=true at product-esb level, to create the distribution pack. At test execution time, the test framework extracts this pack and starts the ESB server to deploy the required artifacts or make the required configuration changes for executing the tests.

Make sure you shutdown any running instances of a WSO2 product or a Axis2 Server instance or any other before running the tests, as it might interrupt with the servers that will be started and stopped during test execution.

Debugging Integration Tests

In order to debug integration tests, do the following :
  1. Navigate to required module or product-esb folder and issue mvn clean install -Dmaven.surefire.debug
  2. Notice the following in console :
    Listening for transport dt_socket at address: 5005
  3. Now click on debug button or Run>Debug in IntellijIDEA or Eclipse (Notice that you need to do a remote debugging here. Refer http://blog.maheeka.me/2014/11/remote-debugging-wso2-esb.html )
  4. You can apply breakpoints, watch expressions as you would in any normal debug scenario

Useful TestNG Tips

These integration tests use TestNG as the test engine.

Test suites and test cases to be run is included in the testng.xml. This testng.xml file can be found at [PRODUCT_ESB]/modules/integration/tests-integration/[relevant_module_folder]/src/tests/resources/testng.xml.

The tests can be included in package level or classes or even method level. Tests can also be excluded in debug Refer [1] to read up on TestNG. Following are some useful tips on TestNG.

1. To define a new test suite, add the following within <suite> tag of testng.xml :
   <test name="[TestSuite name]" preserve-order="true" verbose="2">
        <packages>
            <package name="[package containing tests]"/>
        </packages>
   </test>

set preserver-order to true, if you want to run the tests in the order specified in package or classes.

2. Instead of package, if you want to add classes :
   <test name="[TestSuite name]" preserve-order="true" verbose="2">
        <classes>
            <class name="[fully qualified class name]"/>
            <class name="[fully qualified class name]"/>
            <class name="[fully qualified class name]"/>
        </classes>
    </test>

3. If you want to add specific test methods of a class :
  <test name="[TestSuite name]" preserve-order="true" verbose="2">
        <classes>
            <class name="[fully qualified class name]"/>
                <methods>
                    <include name="[test method name]" />
                    <include name="[test method name]" />
                </methods>
            </class>
        </classes>
    </test>

4. If you want to exclude specific methods of a class :
  <test name="[TestSuite name]" preserve-order="true" verbose="2">
        <classes>
            <class name="[fully qualified class name]"/>
                <methods>
                    <exclude name="[test method name]" />
                    <exclude name="[test method name]" />
                </methods>
            </class>
        </classes>
    </test>

4. You can exclude or include all test methods by using ".*" as regex value for name :
    <exclude name=".*" />
    <include name=".*" />

5. To disable a test suite, use enabled property (true/false) :
    <test name="[TestSuite name]" preserve-order="true" verbose="2" enabled="false">
        ....
    </test>

6. The test suites run in the order specified in the testng.xml. However, the order of the surefire report is different to this

7. Test methods can be identified by @Test annotation in the Java class

8. You can also disable test methods in Java class
    @Test(groups = {"wso2.esb"}, description = "...", enabled = false)
    public void testMethod() {
        .....
    }
Refer next post for notes on writing ESB integration tests.

Reference : 
  1. http://testng.org/doc/documentation-main.html

February 22, 2015

Using Code Formatter for Blogger

Wordpress have a <code> tag to wrap code segments, but this is not so in blogger. So when I migrated from wordpress to blogger, all the formatting and syntax highlighting had to be redone.

I used SyntaxHighlighter [1] for this.

Step 1 :

First step is to add required css and javascript to the blogger template. So whatever template you have chosen, go to your Blogger Home > Template > Edit HTML

Step 2 :

Add the following within <head> tag

<!-- BEGIN SyntaxHighlighter BEGIN -->

<link href='http://alexgorbatchev.com/pub/sh/current/styles/shCoreDefault.css' rel='stylesheet' type='text/css'/>

<link href='http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css' rel='stylesheet' type='text/css'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushBash.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCpp.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCSharp.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCss.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushDiff.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJava.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJavaFX.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJScript.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPerl.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPhp.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPlain.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPython.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushRuby.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushSql.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushVb.js' type='text/javascript'/>

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js' type='text/javascript'/>

<script language='javascript' type='text/javascript'>

    SyntaxHighlighter.config.bloggerMode = true;

    SyntaxHighlighter.all();

 </script>

<!-- END SyntaxHighlighter END -->


Step 3 : Writing blog posts

In your blog posts, go to HTML view and wrap all your code segments with a <pre> tag as below,

<pre class="brush:java">

System.out.println ("Using Syntax Highlighter");

</pre>

Below is the real preview of the above:

System.out.println ("Using Syntax Highlighter");
For more info on SyntaxHighlighter configurations, visit [2]

Reference :
  1. http://alexgorbatchev.com/SyntaxHighlighter/
  2. http://alexgorbatchev.com/SyntaxHighlighter/manual/configuration/
  3. http://patrickwebster.blogspot.com/2009/02/syntaxhighlighter-in-blogger.html

February 13, 2015

Apple Script to copy a file repeatedly


Open Apple Script Editor and add the following script and hit run button.

set theFile to "Macintosh HD:Users:maheeka:original_file:test.xml"
set theDestinationFolder to "Macintosh HD:Users:maheeka:test_des"

repeat 10 times
tell application "Finder"
duplicate theFile to theDestinationFolder with replacing
end tell
delay 2
end repeat


Note :
  • Always use an alias for the file location
  • File path should be separated with : and not /

January 28, 2015

Remove 3rd Party Apps in OSX

Removing apps installed on OSX via App Store is very easy. You just have to go to LaunchPad tap and hold on the icon of the app you want to uninstall until a small cross appears on the top right corner. Click on this cross and you can uninstall the app.

However, for 3rd party apps where you either downloaded a .dmg and installed or however, it is not that straightforward. So in order to do that you can do the following. 

Example : 

Maheekas-MacBook-Pro:~ maheeka$ mdfind -name mackeeper
/Applications/MacKeeper.app
/Users/maheeka/Library/LaunchAgents/com.zeobit.MacKeeper.Helper.plist
/Users/maheeka/Library/Application Support/MacKeeper Helper

This lists down all folders and files in the system with name as "mackeeper". Delete each of these folders and files. 

Local FTP Server on OSX


To start a local FTP server : 
$ sudo -s launchctl load -w /System/Library/LaunchDaemons/ftp.plist

To stop local FTP server : 
$ sudo -s launchctl unload -w /System/Library/LaunchDaemons/ftp.plist

To connect to FTP server : 
Maheekas-MacBook-Pro:~ maheeka$ ftp localhost
Trying ::1...
Connected to localhost.
220 ::1 FTP server (tnftpd 20100324+GSSAPI) ready.
Name (localhost:maheeka): maheeka
331 User maheeka accepted, provide password.
Password: 
230 User maheeka logged in.
Remote system type is UNIX.
Using binary mode to transfer files.

ftp> 

Once connected, most of normal file system related commands will work similarly.


ftp> pwd
Remote directory: /Users/maheeka
ftp> mkdir test
257 "test" directory created.
ftp> cd test
250 CWD command successful.
ftp> mkdir newfolder
257 "newfolder" directory created.
ftp> ls
229 Entering Extended Passive Mode (|||50051|)
150 Opening ASCII mode data connection for '/bin/ls'.
total 0
drwxr-xr-x  2 maheeka  staff  68 Feb  1 10:02 newfolder
226 Transfer complete.
ftp> pwd

Remote directory: /Users/maheeka/test

For a complete reference of ftp commands refer : http://krypted.com/commands/ftp-commands/

To access FTP Server from browser :

Type ftp://username:password@localhost/ on address bar.


Reference : 

November 13, 2014

Remote Debugging WSO2 ESB with IntellijIdea and Eclipse

This post assumes you are familiar with debugging concepts. The remote debugging will be explained using WSO2 ESB product.

What happens in remote debugging is that the debugging information of a running application is communicated with an IDE that has the source of the application. The application is not running from the source, and thus the term "remote" debugging. Similar to any normal debug scenario, where you would be running the application in debug mode from the IDE, you can have breakpoints, watch expressions, etc.

Read more about how remote debugging works in JVM at [1].

Let's proceed on how to debug WSO2 ESB.

Setup the Source for Debugging WSO2 ESB

There are a number of components in WSO2 stack that contribute to the complete WSO2-ESB product. Some of these components are : wso2-synapse, carbon-mediation, wso2-axis2-transport, carbon4-kernel, etc. You can download source of all these components at https://github.com/wso2.

Decide on the component(s) you need to debug, and import them to your preferred IDE (Eclipse or IntellijIdea). To do this :
  1. Clone the required component from the git repository : git clone <repository_url>
  2. Build the component : mvn clean install or mvn clean install -Dmaven.test.skip=true (to skip tests)
  3. To setup as an IDE project : mvn idea:idea (for IntellijIdea) or mvn eclipse:eclipse (for Eclipse)
  4. Add breakpoints as needed
  5. If debugging multiple components, 
    • Eclipse : Import as project to the same workspace and add breakpoints
    • Idea : Import as module to the same window and add breakpoints
    • Optionally, at the time of debug, when debugging takes you to the .class files, you can attach the relevant components as sources with "attach source" option and continue.
Note : 
If you want to debug message flow on ESB, add a breakpoint to receive(org.apache.axis2.context.MessageContext mc) method in org.apache.synapse.core.axis2.ProxyServiceMessageReceiver class in wso2-synapse. This is the start point to the message flow on ESB.

Remote Debugging WSO2-ESB

Step 1 : Start WSO2-ESB in debug mode

$ sh bin/wso2server.sh debug 5005
Please start the remote debugging client to continue...
JAVA_HOME environment variable is set to /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
CARBON_HOME environment variable is set to /Users/maheeka/wso2esb-4.8.1
Listening for transport dt_socket at address: 5005

We have started the ESB in debug mode, and it listens now on port 5005 for a remote debugging client. 

Step 2 : Start Remote Debugging Client (IDE)

IntelliJIdea
  1. Run > Edit Configurations
  2. On the left panel, click on "+" and add a new Remote Configurations. Notice the port to be same as you give when starting ESB.
  3. Then Run > Debug 'wso2-synapse' to start the listener
  4. The application will now run as normal and stop at the breakpoints you have added




Eclipse
  1. Run > Debug Configurations
  2. Add a new Remote Java Application Configuration. Notice the port number to be same as you give when starting ESB.
  3. Click on Debug to start debugging 

November 10, 2014

Building WSO2 ESB 4.8.1 from source in OSX

WSO2 products are all built on top of WSO2 Carbon Kernel. This is the common ground for all products. Turing is the platform release name for WSO2 Carbon 4.2.0 and WSO2 ESB 4.8.1 is based on turing. Not all products of the WSO2 platform does not get released at the same time. Therefore, these products are released in chunks. ESB 4.8.1 comes under chunk 7 of the turing platform. Refer the release matrix to get more info on this [2].
Prerequisites : 
1. Install Java 1.6 and Maven 3
       If you have only 1.7 installed, you will have to install 1.6 on OSX. Unfortunately, there is no direct installation for 1.6. But this sorted out for me  since jdk 1.6 was automatically installed when I installed IntellijIdea
       At the moment, all WSO2 products are compiled on Java 1.6 and is supported up to 1.7. This is why we need to build the source on 1.6.
2. Checkout Source (Orbit, Kernel, Platform)

Building the Source : 


Start building in order Orbit > Kernel > Platform (chunk 07) repos respectively.


To build ESB 4.8.1, you need to build wso2carbon-platform/product-releases/chunk-07 as Platform in above.


The distribution will be created in :

wso2carbon-platform/products/esb/4.8.1/modules/distribution/target/(esb.zip)

When building, since it will take a lot of time, and to avoid unit test failures from building the product, skip unit tests when building.


mvn clean install -Dmaven.test.skip=true


Build ESB only :

If you need to build only ESB, build at wso2carbon-platform/products/esb with mvn clean install -Dmaven.test.skip=true


Another option is to comment out other product modules in platform/trunk/pom.xml and rebuilding in Orbit > Kernel > Platform in order.


Encountered Errors : 
  1. WSS4 maven install fails > You need to compile in Java 1.6
  2. [WARNING] Error injecting: org.apache.maven.reporting.exec.DefaultMavenReportExecutor > Set maven-site plugin version to 3.3 in all places [4]
Environment : 

October 26, 2014

Grep - Find a text in file (Unix/Linux)


To find the text files containing a particular string :

grep -l "maheeka" *.txt

Search in current directory :
grep -lr "maheeka" . 

To view the occurrences of the string in the files, use without "-l" option

Reference:
http://hints.macworld.com/article.php?story=20001105214242629