TransactionManager Debug and Trace events

Posted on by

While working on a project, from time to time, we might request the TransactionManager ‘Debug’ and ‘Trace’ events.

The Debug event is the one created by the org.jpos.transaction.Debug participant and looks like this:

<log realm="debug" at="Sun Feb 12 22:47:39 UYST 2012.717" lifespan="2ms">
  <commit>
    <id>2</id>
    <context>
      <entry key='REQUEST'>
       <isomsg direction="incoming">
         <!-- org.jpos.iso.packager.GenericPackager[cfg/jcard.xml] -->
         <field id="0" value="2100"/>
         <field id="2" value="600933______0011"/>
         <field id="3" value="555555"/>
         <field id="4" currency="858" type="amount" value="0.00"/>
         <field id="7" value="0212224739"/>
         <field id="11" value="000000000003"/>
         <field id="12" value="20120212224739"/>
         <field id="13" value="000212"/>
         <field id="14" value="1001"/>
         <field id="17" value="0212"/>
         <field id="32" value="000001"/>
         <field id="37" value="12104759694 "/>
         <field id="41" value="29110001        "/>
         <field id="42" value="001001"/>
         </isomsg><isomsg id="43">
           <!-- org.jpos.iso.packager.GenericSubFieldPackager -->
           <field id="2" value="jCard Selftest system"/>
           <field id="4" value="Montevideo"/>
           <field id="5" value="MV "/>
           <field id="7" value="UY "/>
         </isomsg>
         <isomsg id="113">
           <!-- org.jpos.iso.packager.GenericSubFieldPackager -->
           <field id="2" value="106"/>
         </isomsg>
 
      </entry>
      <entry key='SS'>JCARD</entry>
      <entry key='TXNNAME'>100.55</entry>
      <entry key='SOURCE'>org.jpos.iso.channel.CSChannel@71e13a2c</entry>
      <entry key='PROFILER'>
       <profiler>
         open [0.6/0.6]
         prepare-response [6.0/6.6]
         close [1.0/7.6]
         end [15.3/15.3]
       </profiler>
      </entry>
      <entry key='TIMESTAMP'>Sun Feb 12 22:47:39 UYST 2012</entry>
      <entry key='TXNMGR'>txnmgr</entry>
      <entry key='DB'>org.jpos.ee.DB@7a315068</entry>
      <entry key='SWITCH'>100.55 (notsupported prepareresponse close sendresponse)</entry>
      <entry key='RC'>invalid.request</entry>
      <entry key='EXTRC'>Unsupported transaction</entry>
      <entry key='IRC'>1015</entry>
      <entry key='RESPONSE'>
       <isomsg direction="outgoing">
         <!-- org.jpos.iso.packager.GenericPackager[cfg/jcard.xml] -->
         <field id="0" value="2110"/>
         <field id="2" value="600933______0011"/>
         <field id="3" value="555555"/>
         <field id="4" currency="858" type="amount" value="0.00"/>
         <field id="7" value="0212224739"/>
         <field id="11" value="000000000003"/>
         <field id="12" value="20120212224739"/>
         <field id="13" value="000212"/>
         <field id="14" value="1001"/>
         <field id="17" value="0212"/>
         <field id="32" value="000001"/>
         <field id="37" value="12104759694 "/>
         <field id="39" value="0030"/>
         <field id="41" value="29110001        "/>
         <field id="42" value="001001"/>
         </isomsg><isomsg id="43">
           <!-- org.jpos.iso.packager.GenericSubFieldPackager -->
           <field id="2" value="jCard Selftest system"/>
           <field id="4" value="Montevideo"/>
           <field id="5" value="MV "/>
           <field id="7" value="UY "/>
         </isomsg>
         <field id="63" value="FORMAT ERROR"/>
         <isomsg id="113">
           <!-- org.jpos.iso.packager.GenericSubFieldPackager -->
           <field id="2" value="106"/>
         </isomsg>
         <field id="123" value="Unsupported transaction"/>
 
      </entry>
    </context>
  </commit>
</log>

It basically dumps the Context. On the other hand, the Trace event usually comes after the Debug (provided the TransactionManager’s trace property is set to true) and looks like this:

<log realm="org.jpos.transaction.TransactionManager" at="Sun Feb 12 22:47:39 UYST 2012.725" lifespan="23ms">
  <debug>
    txnmgr-1:2
            prepare: org.jpos.jcard.PrepareContext NO_JOIN
            prepare: org.jpos.jcard.CheckVersion READONLY NO_JOIN
            prepare: org.jpos.transaction.Open READONLY NO_JOIN
            prepare: org.jpos.jcard.Switch READONLY NO_JOIN
      groupSelector: notsupported prepareresponse close sendresponse
            prepare: org.jpos.jcard.NotSupported NO_JOIN
            prepare: org.jpos.jcard.PrepareResponse NO_JOIN
            prepare: org.jpos.transaction.Close READONLY
            prepare: org.jpos.jcard.SendResponse READONLY
            prepare: org.jpos.jcard.ProtectDebugInfo READONLY
            prepare: org.jpos.transaction.Debug READONLY
             commit: org.jpos.transaction.Close
             commit: org.jpos.jcard.SendResponse
             commit: org.jpos.jcard.ProtectDebugInfo
             commit: org.jpos.transaction.Debug
    head=3, tail=3, outstanding=0, active-sessions=2/2, tps=0, peak=0, avg=0.00, elapsed=22ms
    <profiler>
      prepare: org.jpos.jcard.PrepareContext [0.0/0.0]
      prepare: org.jpos.jcard.CheckVersion [0.0/0.0]
      prepare: org.jpos.transaction.Open [0.5/0.6]
      prepare: org.jpos.jcard.Switch [0.0/0.6]
      prepare: org.jpos.jcard.NotSupported [0.1/0.7]
      prepare: org.jpos.jcard.PrepareResponse [5.8/6.6]
      prepare: org.jpos.transaction.Close [0.0/6.6]
      prepare: org.jpos.jcard.SendResponse [0.0/6.6]
      prepare: org.jpos.jcard.ProtectDebugInfo [0.0/6.7]
      prepare: org.jpos.transaction.Debug [0.0/6.7]
       commit: org.jpos.transaction.Close [1.0/7.7]
       commit: org.jpos.jcard.SendResponse [4.3/12.0]
       commit: org.jpos.jcard.ProtectDebugInfo [0.2/12.3]
       commit: org.jpos.transaction.Debug [9.3/21.7]
      end [22.8/22.8]
    </profiler>
  </debug>
</log>

jPOS Maven Archetype

Posted on by

If you use Maven, there’s an extremely easy way to create a jPOS application now:

mvn archetype:generate \
  -DarchetypeGroupId=org.jpos \
  -DarchetypeArtifactId=jpos-archetype \
  -DarchetypeVersion=1.8.2 \
  -DarchetypeRepository=http://jpos.org/maven

If you don’t want to use the remote archetype, you can always ‘mvn install’ jPOS source code and then:

mvn archetype:generate -DarchetypeCatalog=local

UPDATED to use 1.8.2 instead of 1.8.1-SNAPSHOT
UPDATED to use ‘archetypeRepository’ instead of ‘DremoteRepositories’

jPOS Performance

Posted on by

First Data NOC

When companies — specially start-ups — ask us regarding jPOS performance, they usually tell us they need to support a massive load in the 1000+ TPS range. In order to give some perspective to these requirements, we always exchange some industry information with jPOS power developer Andy whose company manage some jPOS systems reaching 200M transactions per month, and today Andy sent me a very nice link (via OLS’ Ron Barker) about one of the largest acquirers in the world, during a really busy day, requiring 988 TPS, the important part:

“The record day on Nov. 26 translated to about 988 payment authorizations per second.”

So I thought it would be nice to start having some stable performance tests here and there in jPOS, so we can have something to measure where we are now, and even if it’s really fast for most applications, attempt to improve it.

Because many developers use jPOS just to pack and unpack ISO-8583 messages, I thought it would be nice to start by measuring where we are now in the pack/unpack department. I believe there’s plenty of room for optimization, but I hope you’ll agree that we are in pretty good shape.

I’ve just added a performance test to PackagerTestCase.java to try:

  • 100000 pack operations
  • 100000 unpack operations
  • 100000 pack/unpack operations
  • 100000 update, pack, and unpack operations

and here are the results:

<profiler>
  PACK tps=31531, peak=31531, avg=25156.00 [3779.2/3779.2]
  UNPACK tps=32371, peak=37026, avg=29081.00 [3047.9/6827.2]
  PACK/UNPACK tps=16605, peak=16634, avg=16144.25 [6254.2/13081.5]
  UPDATE/PACK/UNPACK tps=10862, peak=10862, avg=10815.16 [10430.9/23512.5]
  end [23512.6/23512.6]
</profiler>

We are operating with a pretty heavy message, way bigger than a real-life ISO-8583 message:

<isomsg>
  <field id="0" value="0800"/>
  <field id="2" value="1234567890123456789"/>
  <field id="3" value="000000"/>
  <field id="4" value="000000000100"/>
  <field id="5" value="000000000200"/>
  <field id="6" value="000000000300"/>
  <field id="7" value="0611191024"/>
  <field id="8" value="12345678"/>
  <field id="9" value="12345678"/>
  <field id="10" value="12345678"/>
  <field id="11" value="000001"/>
  <field id="12" value="010203"/>
  <field id="13" value="0611"/>
  <field id="14" value="1001"/>
  <field id="15" value="0112"/>
  <field id="16" value="0112"/>
  <field id="17" value="0611"/>
  <field id="18" value="1234"/>
  <field id="19" value="858"/>
  <field id="20" value="032"/>
  <field id="21" value="400"/>
  <field id="22" value="000"/>
  <field id="23" value="001"/>
  <field id="24" value="003"/>
  <field id="25" value="00"/>
  <field id="26" value="01"/>
  <field id="27" value="1"/>
  <field id="28" value="000000100"/>
  <field id="29" value="000000200"/>
  <field id="30" value="000000300"/>
  <field id="31" value="000000400"/>
  <field id="32" value="123456"/>
  <field id="33" value="654321"/>
  <field id="34" value="ABCDEFGHIJKLMNOPQRSTUVWXYZ01"/>
  <field id="35" value="1234567890123456=12345678901234567890"/>
  <field id="36" value="THE QUICK BROWN FOX JUMPED OVER THE LAZY DOG"/>
  <field id="37" value="123456789ABC"/>
  <field id="38" value="123456"/>
  <field id="39" value="00"/>
  <field id="40" value="ABC"/>
  <field id="41" value="0001ABCD"/>
  <field id="42" value="123456789012345"/>
  <field id="43" value="JPOS CARD ACCEPTOR / WWW.JPOS.ORG       "/>
  <field id="44" value="jPOSjPOSjPOSjPOSjPOSjPOS"/>
  <field id="45" value="JPOSJPOSJPOSJPOSJPOSJPOS"/>
  <field id="46" value="ADDITIONAL DATA - ISO"/>
  <field id="47" value="ADDITIONAL DATA - NATIONAL"/>
  <field id="48" value="ADDITIONAL DATA - PRIVATE"/>
  <field id="49" value="032"/>
  <field id="50" value="032"/>
  <field id="51" value="032"/>
  <field id="52" value="0001020304050607" type="binary"/>
  <field id="53" value="1234567890123456"/>
  <field id="54" value="000000001000000000002000"/>
  <field id="55" value="ABCD"/>
  <field id="56" value="EFGH"/>
  <field id="57" value="IJKL"/>
  <field id="58" value="MNOP"/>
  <field id="59" value="QRST"/>
  <field id="60" value="UVWX"/>
  <field id="61" value="YZ01"/>
  <field id="62" value="A002"/>
  <field id="63" value="ABCDDCBA"/>
  <field id="64" value="CAFEBABECAFEBABE" type="binary"/>
  <field id="90" value="000000000000000000000000000000000000000000"/>
</isomsg>

The last test is the more realistic one, because we tweak the message by changing some fields (i.e. 7,11,12,13), then pack and unpack.

In most jPOS applications, we receive a message, change a few fields, and send it back, for instance, a typical server will reply by just changing a few fields (i.e. 39, 38) and eventually resetting a few others. I think there’s tremendous room for optimization by caching the binary image of the packed message at the field level, avoiding a re-pack unless the field has changed.

Now that we know where we are, I hope to start testing some optimizations soon. You’re encourage to do so too.

BTW, these figures were taken on an MBP, it would be nice to have some feedback from runs on larger servers.

jPOS 1.8 series

Posted on by

Starting in jPOS 1.8, we are going to manage jPOS builds using Maven.

There’s a mvn branch soon to be merged with master.

It seems 1.7.2 will be the last version of the old series managed by ‘ant’, will jump to 1.8.0 soon.

You can try 1.8.2 by adding to your pom.xml:

    <repository>
      <id>jpos</id>
      <name>jPOS Central Repository</name>
      <url>http://jpos.org/maven</url>
      <layout>default</layout>
    </repository>
 
    <dependency>
      <groupid>org.jpos</groupid>
      <artifactid>jpos</artifactid>
      <version>1.8.2</version>
    </dependency>

1.7.2 has already been committed to jPOS EE.

We’ll continue with the old module-based build system on jPOS-EE for a while.

UPDATE: changed 1.8-SNAPSHOT to 1.8.0
UPDATE: changed 1.8-SNAPSHOT to 1.8.2

JRuby support

Posted on by

/by apr/

I’ve spent a great time listening to great talks at RubyConf Uruguay organized by Cubox. Now I totally understand when they say “Old School Teamwork Fun People, Serious pros: The mix you’re looking for” in their website. It is exactly that.

I was delighted to listen to Charles Nutter (JRuby’s author) talk but was sad with this tweet from him a few hours ago:

Headius Sad

So I tried latest JRuby on latest jPOS, it took just a few minutes. I’ve added an optional ‘jruby’ module to jPOS-EE (just depends on the ‘commons’ module) and you can now deploy a QBean that looks like this:

<ruby class='org.jpos.jruby.Ruby'>
  puts "Hello jRuby!"
</ruby>

And that’s all there is to it :)

If you want to try this in a new small project, you can follow the HOWTO in jPOS Project Guide around page 35, just add the ‘commons’ and ‘jruby’ module as svn:externals, and you’d be ready to go.

And the output, you guess it:

Hello jRuby!

Mr. Nutter, Welcome to this southern Montevideo and beware with the dogs here :)

jPOS-EE Things

Posted on by

Inspired by this talk, I got reminded to play with EAVs in jPOS-EE.

I’ve created a jPOS-EE modules called “things” that I believe it’s a good thing to have in most applications.

I don’t think this is a one-size-fits-all solution, you probably don’t want to store your core tables (such as the transaction log, miniGL stuff, whatever) using this open schema approach, but for everything else, having this open schema can save your day (in the same way a ‘flags’ field in most tables can help at some point).

Using it is very simple, you create a “thing” using the ThingsManager:

   DB db = new DB();   
   ThingsManager mgr = new ThingsManager(db);
   Thing t = mgr.create("MyThing");

Then you can put Strings, Longs, Dates, Timestamps, long Strings and BigDecimals to your thing, i.e:

   t.put ("AString",  "This is a string");
   t.put ("ALong", Long.MAX_VALUE);
   t.put ("MyInteger", i);
   t.put ("Date", d);
   t.put ("Timestamp", new Timestamp(now));
   t.put ("BigDecimal", ONE_THOUSAND);
   t.putText("Text", "The quick brown fox is brown and the dog is lazy, and jumps.");

The ThingsManager has some handy methods already, and we’ll add more as the need arise.

public List getAll (String type) ;
public Thing getLast (String type);
public List listByStringName (String type, String name);
public List listByStringValue (String type, String value);
public List listByStringNameValue (String type, String name,String value);
public List listByTextName (String type, String name);
public List listByTextValue (String type, String value);
...
...

Some immediate use comes to mind, we could use this to provide a more type-safe, versionable version of our existing SysConfig table (i.e. we could have a thing called ‘SysConfig’ and keep previous versions as ‘SysConfig;1′, ‘SysConfig;2′, etc. (or another numbering scheme if you find this one too VMS-eske). It’s also good to handle extremely proprietary stuff such as fee configuration, currency exchange, etc. found in many jPOS applications.

The things module is available in jPOS-EE as of r307

Q2 transient services

Posted on by

In jCard and jPTS we use the concept of Stations, we have Source Stations (SS), Destination Stations (DS), Monitoring Stations (MS), Control Stations (CS) and Cryptographic Stations (HSMS), etc.

Every station usually requires a handful Q2 services (QBeans), such as a MUX or MUXPool, one or more ChannelAdaptors with their filters, LogonManager, KeyExchangeManagers, eventually an independent logger, etc.

You can configure those manually for a small set of stations, you can also use some scripts, but in order to support a large number of stations, and to easily manage them from the UI, we use a database to keep their basic configuration (station type, host, port, timeouts, etc.) and then use these new transient services provided by Q2.

When Q2 starts, it’s create an unique transient UUID, that FYI is displayed by the SystemMonitor task:

<log realm="org.jpos.q2.qbean.SystemMonitor" at="Sat Jul 24 20:14:14 UYT 2010.495">
  <info>
    <release>jPOS 1.6.9 r2950</release>
    <instance>e424833b-c2c1-4f8b-b743-8a69271912a2</instance>
    <uptime>00:00:00.170</uptime>
  </info>
   ...
   ...
</log>

When you deploy a QBean using Q2′s deployElement method (which is now public), you can flag that qbean as ‘transient’ (there’s a boolean parameter). In that case, Q2 will remove the file on exit, but just in case the system crashes, it also adds a ‘transient’ attribute to the QBean, i.e:

<server class="org.jpos.q2.iso.QServer" logger="Q2"  instance="e424833b-c2c1-4f8b-b743-8a69271912a2">
  ...
  ...
</server>

The next time it runs, Q2 will generate a new instance ID, so in the rare situation where an old deployment descriptor is present in the deploy directory, it will be ignored and deleted (as it should have been removed at the previous exit).

Q2.deployElement is a handy method that requires a JDom Element. If you are not a member of the JDom church, you can always create your QBean manually, you can get to know Q2′s instance ID by calling its getInstanceId() method that gives you an UUID,.

Note: when you deploy a bundle using Q2 –config=/path/to/your/bundle, Q2 now flags the exploded descriptors as transient.

This is available as of jPOS 1.6.9 r2950

jPOS 1.6.8 has been released

Posted on by

Just a quick note to let you know that jPOS 1.6.8 has been released (ChangeLog).

This is a maintenance release that fixes several bugs and adds some reasonable defaults that can avoid some problems (i.e.TransactionManager’s paused-timeout default issue).

jPOS-EE users just ‘svn update’ and you’ll get it from modules/jpos/lib/jpos.jar (r290).

Technology XYZ

Posted on by

/by apr/

There are several recurring questions that jPOS new users keep asking:

  • I want to build jPOS in Netbeans/Eclipse/IDEA
  • I want to log to log4j
  • I want to run it in Tomcat/Spring/Websphere/Weblogic/JBoss (note the mix of technologies here)

Here is how I read them:

  • I want to build jPOS in my IDE because I don’t have a clue about compiling a java program using ant and I’m lazy to learn something new. I don’t even know how to ‘cd’ to the jPOS directory.
  • I know how to configure log4j and don’t want to learn how to configure other stuff. jPOS comes pre-configured to use a very nice PCI-compliant log, but I’m so lazy that I don’t know that, so I ask how to use it with log4j.
  • I don’t know how to start a Java application as a daemon/service, but my Tomcat/Spring/Websphere somehow is already running in my server (the previous admin left it running before quitting), so I want to run jPOS inside one of those.

So people don’t care about the business logic, about why they need jPOS, they just want to have something running in their environment, even if the environment is not the best choice to run their jPOS applications.

jPOS is a first class application that will process transactions worth millions of dollars for you. You should consider asking us what’s the best environment, the best DB, the OS to run jPOS instead of forcing it to run in weird ways.

Amen.