148 posts tagged with "jpos"

The jPOS TransactionManager context, when it comes to traverse all participants, is quite promiscuous, as promiscuous as the global state of any application. It relies on the participants to do the right thing and don't step over each other, by mistake, or on purpose.

As applications grow and we move from a handful participants to several dozens, or even hundreds, it becomes very useful to put some constraints on what a participant can or can't do related to its Context.

On large jPOS applications, created by disparate teams, the TransactionManager configuration file is usually an excellent self-documenting starting point in order to figure out which transactions are supported, which participants are involved and how they are configured.

In addition, it is common for a participant’s implementation, before doing its job, to check for the presence in the Context of its needed input variables. CheckFields for example, needs a REQUEST object. SendResponse, needs a RESPONSE object.

To solve this, we've introduced a new required element, verified by the TransactionManager before calling the participant's prepare method, that looks like this:

<participant class="org.jpos.transaction.CheckFields">
  <requires>REQUEST</requires>
</participant>

In this case, the TM checks for the presence of a REQUEST entry in the Context. If it’s not present, the transaction aborts right away (as if the participant had returned ABORTED).

This is just syntactic sugar, as the participant can check (and actually check) that, but there's a little twist.

The participant does not receive the whole context, it just receives a clone of it with JUST the entries described in the <require> element (which BTW, can be a comma separated list).

In addition to the require element, we support an optional element. i.e.:

<participant class="org.jpos.transaction.CheckFields">
  <requires>REQUEST</requires>
  <optional>TIMESTAMP, TIMEZONE</optional>
</participant>

You get the idea, if the optional TIMESTAMP and TIMEZONE entries are present, they would get copied to the mini Context used to call the participant.

A similar approach is used in order to selectively get the output of a participant. We use the provides element like this:

<participant class="org.jpos.transaction.CreateDSRequest">
  <requires>REQUEST</requires>
  <provides>RESPONSE</provides>
</participant>

The TransactionManager, who has access to both the original Context and the miniContext offered to the participant, merges back ONLY the entries listed in the provides method.

This is an experimental preview feature implemented in jPOS 3.0 (next). It needs to be battle tested in order to see if it makes sense to keep it, or it's just a useless idea. If we like it, and we continue supporting it, we may choose to enforce it on every participant. In a second phase we can define a TM-level promiscuousproperty so that if one chooses the old behavior, you need to set it true, either globally at the TM config level, or at the participant level, perhaps with a promiscuous="true" attribute.

jPOS 2.1.8 has been released. New development version is 2.1.9-SNAPSHOT and jPOS 3.0.0 (aka 'next').

See ChangeLog for details.

We will continue to maintain the 2.x.x series for a long while, so you can consider the 2.x.x series an LTS version, but all the fun and new features will happen in 3.x.x.

Typical jPOS applications have many similar servers, muxes, and channels. Sometimes hundreds of them.

In order to simplify its configuration Q2 now supports templates that can be launched from a template deployer, or programmatically using Q2's deployTemplate method.

A template is identical to a regular QBean XML descriptor, and can be placed anywhere in the filesystem, or classpath (including dynamic classpath) that has a special keyword called __PREFIX__ that are replaced at deploy time.

Q2's deployTemplate method has the following signature:

deployTemplate(String resource, String filename, String prefix);

• resource is the resource location (e.g.: templates/channel.xml). If the resource starts with jar:, Q2 fetchs it from the classpath (e.g.: jar:META-INF/q2/templates/channel.xml).
• filename is the QBean descriptor filename, e.g.: 10_acme_channel.xml
• prefix is the Environment prefix (see below).

Imagine you have an environment (cfg/default.yml) file that looks like this:

    acme:
        host: 192.168.1.1
        port: 8000
    emca:
        host: 192.168.1.2
        port: 8001

In a regular deployment, you could have two channel configurations like this:

    10_channel_acme.xml
    
    <channel-adaptor name="acme-channel-adaptor" ...>
      <channel class="${acme.channel}" packager=...>
        <property name="host" value="${acme.host}" />
        <property name="port" value="${acme.port}" />
        ...
        ...
      </channel>
      <in>acme-send</in>
      <out>acme-receive</out>
    </channel-adaptor>
    
    10_channel_emca.xml
    
    <channel-adaptor name="emca-channel-adaptor" ...>
      <channel class="${emca.channel}" packager=...>
        <property name="host" value="${emca.host}" />
        <property name="port" value="${emca.port}" />
        ...
        ...
      </channel>
      <in>emca-send</in>
      <out>emca-receive</out>
    </channel-adaptor>

With templates, you can have a single template like this somewhere in your filesystem or classpath (say templates/channel.xml).

<channel-adaptor name="__PREFIX__-channel-adaptor" 
    enabled="${__PREFIX__.enabled:false}"
    class="org.jpos.q2.iso.ChannelAdaptor" 
    logger="${__PREFIX__.logger:Q2}">
 <channel class="${__PREFIX__.channel}" 
         logger="${__PREFIX__.logger:Q2}" 
       packager="${__PREFIX__.packager:org.jpos.iso.packager.GenericPackager}">
  <property name="host" value="${__PREFIX__.host}" />
  <property name="port" value="${__PREFIX__.port}" />
  <property name="header" value="${__PREFIX__.header}" />
  <property name="connect-timeout" value="${__PREFIX__.timeout:5000}" />
  <property name="packager-config" value="${__PREFIX__.packagerConfig}" />
 </channel>
 <in>__PREFIX__-send</in>
 <out>__PREFIX__-receive</out>
 <reconnect-delay>${__PREFIX__.reconnect:10000}</reconnect-delay>
 <wait-for-workers-on-stop>${__PREFIX__.wait:yes}</wait-for-workers-on-stop>
</channel-adaptor>

and perhaps a mux.xml connected to it:

----
<mux name="__PREFIX__" class="org.jpos.q2.iso.QMUX" 
    enabled="${__PREFIX__.enabled:false}"
    logger="${__PREFIX__.logger:Q2}" realm="__PREFIX__">
  <key>${__PREFIX__.mux.key:41,11}</key>

 <in>__PREFIX__-receive</in>
 <out>__PREFIX__-send</out>

 <ready>__PREFIX__-channel.ready</ready>
</mux>
----

Then you can deploy a QBean descriptor like this:

<templates>
    <template resource="templates/channel.xml" descriptor-prefix="10_channel_">
        acme,emca
    </template>
    <template resource="templates/mux.xml" descriptor-prefix="20_mux_">
        acme,emca
    </template>
</templates>

The special text __PREFIX__ is replaced by Q2 in each file using the prefix acme and emca, so the properties:

  <property name="host" value="${__PREFIX__.host}" />
  <property name="port" value="${__PREFIX__.port}" />

gets converted to

  <property name="host" value="${acme.host}" />
  <property name="port" value="${acme.port}" />

Very simple, it's just configuration sugar, but comes very handy in deployments with hundreds of services.

This is available in jPOS Next (series 3.x.x) starting in 3.0.0-SNAPSHOT availble in jPOS Maven repo, as well as Github next branch.

The TransactionManager's participants provide easy ways to control per transaction timeouts. One can add a TIMESTAMP entry at the beginning of the transaction, and then have each participant control how it's going, so to speak. If the transaction has spent more than a threshold amount of time, it's better to abort the transaction right away because the client (i.e. POS device) probably has expired the transaction already and it's not waiting for our response anymore.

We can do that on each participant, or you can have a general-purpose timeout participant that you can place here and there.

But because this is a common use case, to encourage its use, in jPOS Next the TransactionManager honors a couple of new attributes at the participant level:

• timeout
• max-time

e.g.:

  ...
  ...
  <participant class="org.jpos.jcard.CheckFields">
   <property name="mandatory" value="PCODE,7,11,12,AMOUNT,PAN,41" />
  </participant>
  <participant class="org.jpos.jcard.SelectCurrency" />
  <participant class="org.jpos.jcard.CreateTranLog" timeout="1000" max-time="5000">     
   <property name="capture-date" value="capture-date" />
   <property name="checkpoint"   value="create-tranlog" />
   <property name="node"         value="${jcard.node:99}" />
  </participant>
  ...
  ...

In this example, three things can happen:

• if, by the time the TM reaches the CreateTranLog participant, the transaction has taken more than 5000ms thus far, the participant is not called, and the transaction continues with the ABORTED route.
• If the partitipant takes more than 1000ms to process, even if it has returned PREPARED, we coerce the response to ABORTED.
• If max time, at the end of the participant is greater than 5000ms, we also coerce the response to ABORTED.

jPOS has evolved based on simple principles:

• Latest version is always production ready
• We don't rock the boat. All changes are backward compatible.

When Java started its new release cadence, we kept the baseline source compatibility to Java 8, while making sure jPOS was able to run on newer JDKs, but that prevented us to take advantage of new language features. There were features—like a long time waiting for multi-line Strings—that were nice to have, but one could still live without, but then there was one very interesting one for us, Project Loom (formerly fibers) that almost made it to Java 17, that we want to test at some large sites along with the TransactionManager.

Sealed classes, Records, Switch pattern matching, being prepared for JEP411 that will deprecate the Security Manager are all things we want to play with.

In addition, there are some changes we want to try in jPOS that would certainly rock the boat, related to audit logging (yes, we want to be agreegator/JSON/ELK friendly), using Strings instead of Integers in ISOMsg keys (so one doesn't have to create an arbitrary mapping on EMV related messages), Fluent APIs to create components for those that hate XML (funny, people find XML configs archaic but still love pom.xml), in-memory deployments, per-participant profiling on the TransationManager, etc.

In addition, we need to get rid of some baggage. The compat_1_5_2 module will remain available in the repo, as a reference, but it doesn't have to exist in the main tree. Some AWT/Swing based classes may go back to jPOS-EE as a module, and we may have a sibling module using JavaFX in order to see if it takes traction (JavaFX looks promising).

The CLI is up for a refactor. It's super useful to implement jPOS commands, but the API, initially very tied to JLine has proven to be kind of problematic once we added SSH support. We can do better on that side.

So... here is what we are planning to do:

• jPOS 2.x.x series will become an LTS version with just security patches.
• All the fun will happen in jPOS 3.x.x series, that currently has a baseline on Java 17 (but that may change if our Loom tests go well and we need it as a baseline).

For the time being, there's a next branch (already available in Github) that builds jPOS 3.0.0-SNAPSHOT.

The master branch currently has jPOS 2.1.8-SNAPSHOT but once we release 2.1.8, we'll switch next to become master (or take the opportunity to rename it to main) and use 2.1.8 as the forking point for an LTS v2 (or similar name) branch.

jPOS 2.1.7 has been released. New development version is 2.1.8-SNAPSHOT.

See ChangeLog for details.

Work has been started toward jPOS 3.0.0 targeting Java 17 as a baseline.

We will continue to maintain the 2.x.x series for a long while, so you can consider the 2.x.x series an LTS version, but all the fun and new features will happen in 3.x.x.

jPOS-EE 2.2.7 has been released.

New development version is 2.2.8-SNAPSHOT.

ChangeLog here.

jPOS 2.1.6 has been released, new development version is now 2.1.7-SNAPSHOT

Please see the ChangeLog.

See Resources Page for details.

jPOS 2.1.5 has been released, new development version is now 2.1.6-SNAPSHOT

Please see the ChangeLog.

See Resources Page for details.

/by Barzilai Spinak a.k.a. barspi/

Recently we spent two or three days hunting down a bunch of weird behaviors on a client's project.

It was a typical jPTS setup, and we were testing a Source Station (SS), talking to the central jPTS, talking to a destination station (DS). The different subsystems were chained together with the standard set of QueryHost + QMUX + ChannelAdaptor pointing to the QServer down the line.

The client wanted to do some stress load testing, and was using JMeter ^*^ with the jPOS-based ISO-8583 plugin (sending a very high load of messages to the SS).

Despite being a "standard setup", there were several custom features that were giving us all kinds of trouble (it's never a standard setup, is it?). The initial implementation was performing very badly with respect to transactions per second (TPS), and also losing many transactions due to timeouts. We had to isolate the different sources of the problems (extra queries to a very slow parallel database, a DB connection pool too small for the expected concurrency, an HSM that would choke when being hit with some ridiculously high number of TPS...). Things got much better after some refactoring, but we still got many responses with a timeout error code, or just dropped transactions, for which we never got a response. When analyzing the logs further, we saw some warnings from the multiplexers about duplicate keys, and we detected that it was better to use a different key set than QMUX's default 11, 41 (the terminal id, field 41, would often be constant, so we decided to add field 37, the Retrieval Reference Number, to the key set).

But still, many transactions were timing out... until we detected something even stranger. For example, RRN's, or other fields, that would change in the response, returning a value that belonged to a different request. Our ISOMSg's were having some weird genetic recombination!

Now, we should clarify that this system was developed mostly by the client, so we weren't familiar with all the details of their code and logic hidden in some of their custom classes. We were told that the DS was supposed to talk to a Mastercard remote node, but that at the moment it only had some "autoresponder simulator" in order to perform these tests. We imagined a simple RequestListener that just changed and added a couple of fields, with a standard success response in field 39.

In reality, the "autoresponder simulator", was a full transaction manager with several participants, one of which was a not-so-simple equivalent to the following (here, extremely simplified, for clarity and brevity)

public class SomeParticipant implements TransactionParticipant, Configurable {
    private String requestKey;              // GOOD
    private ISOMsg theMsg;             	    // BAD; DON'T DO!

    private void readRequestFromContext(Context ctx) {
(1)     theMsg= ctx.get(requestKey);  		// BAD; DON'T DO!
        // ...manipulate theMsg...
    }
    
    protected int prepare(long id, Serializable context) throws Exception {
        Context ctx = (Context)context;

(2)     readRequestFromContext(ctx);  		// OOPS!
        
(3)     theMsg.set(38, getApprovalCode());	// is theMsg still the same object
(4)     theMsg.set(39, getResponseCode());  // ...and now?

(5)     ctx.put(Constants.DS_RESPONSE, theMsg);	// what about here?

        return PREPARED | NO_JOIN;
    }

    public void setConfiguration(Configuration cfg) throws ConfigurationException {
        requestKey = cfg.get("request-key", "REQUEST");  // GOOD
    }
}

Can you spot the problem? The participant is using an instance variable, called theMsg, to store a reference to the request message in line (1). And what's wrong with that? As explained in Chapter 9 of the jPOS Programmer’s Guide, the TransactionManager creates only one instance of each participant, but there may be multiple concurrent transactions (i.e. TransactioManager sessions) running. So, by using an instance member variable to store the value retrieved in line (1), each concurrent transaction (thread) being processed may be overwriting that value with a different ISOMsg. In fact, at each step where we use theMsg, such as in lines (2), (3), and (4), we could be working with a different ISOMsg!

Also, notice how it's perfectly OK to use instance variables to store configuration options for this participant. These are usually read at the time of participant instantiation and initialization, and set in the setConfiguration() method. Those values shouldn't normally change during the life of the participant.

So, what should we do? Always use either local variables, or pass the values as method arguments, or store them in the Context object and retrieve them from there.

Never use the instance variables of a participant to store transaction state, or you may incur in hard-to-debug race conditions

(*) We may have a blog post about using JMeter with jPOS in the near future