Skip to main content

151 posts tagged with "jpos"

View All Tags

Config Smell

· 5 min read
Alejandro Revilla
jPOS project founder

jPOS configuration is very flexible, but that flexibility can sometimes lead to confusion.

We have two main categories:

  • Build-time static token replacement (jPOS build target).
  • Runtime environments.

Build-time static token replacement

This is described in detail in the Configuration Tutorial, but the basic idea is that at build time, when you call gradle installApp or gradle dist, you have the ability to define a compile-time target.

The default target is called devel and reads an optional property file called devel.properties.

If your devel.properties file looks like this:

ABC=AlphaBravoCharlie

And then you have a QBean configuration like this:

<property name="ABC" value="@ABC@" />

The @ABC@ will be expanded to AlphaBravoCharlie in the final file that will land in the build directory.

If you go to build/install/xxx/deploy/yourxml.xml, you’ll see something like this:

<property name="ABC" value="AlphaBravoCharlie" />

The jPOS build system performs that replacement in XML files in the deploy directory, as well as in configuration files (.cfg, .properties) in the cfg directory.

A typical use case, very useful during development is to define a database name.

You can have a devel.properties file with tokens like this:

dbhost=localhost
dbname=jposee
dbuser=jpos
dbpass=password
dbport=5432

And then have a db.properties file (in src/dist/cfg/db.properties) that looks like this:

connection.url=jdbc:postgresql://@dbhost@:@dbport@/@dbname@

When you call gradle installApp, you’ll see that the destination db.properties file in the build/install/xxx/cfg directory will look something like this:

connection.url=jdbc:postgresql://localhost:5432/jpos

When you run locally, in development mode, your devel.properties file, which happens to be the default one, is the one providing tokens for replacement.

But you can call gradle -Ptarget=ci installApp. In that case, tokens get read from a file called ci.properties that you can use in your continuous integration environment.

CONFIG SMELL 1

If you have some handy tokens in the devel.properties file to make configuration tweaks for a development versus continuous integration target, or you have a local, test, or stresstest environment, that’s fine. However, if you have a prod target, there might be a configuration smell. It can be acceptable, but beware that you are not leaking sensitive information such as database names, URLs, access tokens, etc.

prod.properties should probably be reviewed by your security team.

Runtime Environments

The runtime environments are explained here, but the basic idea is that you can define a variable either in a YAML file, a Java property, or an OS environment variable and use it as part of the configuration using property expansion. Imagine you have an OS environment variable like this:

export DB_PASSWORD=mysupersecurepassword

Then you can configure a QBean, or config file with something like:

<property name="dbpass" value="${db.password}" />

When you call cfg.get("dbpass") from your QBean or participant implementation, you will get the string "mysupersecurepassword".

There are variations you can use, such as defaults like ${db.password:nopassword}, in which case, if the DB_PASSWORD OS environment variable is not available or not exported, a call to cfg.get("dbpass") would return nopassword.

From the jPOS Programmer’s Guide Section 3.3 (Configuration)

SimpleConfiguration recognizes and de-references properties with the format: ${xxx} and searches for a system property, or operating system environment variable under the xxx name. As a fallback mechanism of last resort, the property can be resolved from an environment file, in YAML or properties file syntax, found in the cfg directory (default filename default.yml or default.cfg, which can be overridden by the jpos.env system property).

You can add a default value within the expression itself, using the : (colon) separator. For example ${xxx:def_value}. If the property is not found, the default value will be used.

The format $sys{xxx} de-references just from system properties, $env{xxx} just from the operating system environment, and $cfg{xxx} just from the environment file (the.

In the rare case where a value with the format ${...} is required, the $verb{${...}}format (verbatim) can be used.

In addition, a property named xx.yy.zz can be overridden by the environment variable XX_YY_ZZ (note that dots are replaced by underscore, and property name is converted to uppercase.

Things start to get confusing when we hit Config Smell 2:

CONFIG SMELL 2

We often encounter situations where a build-time target configuration looks like this:

dbpass=${db.password}

and then the dbpass property in a given QBean or configuration file looks like this:

<property name="dbpass" value="@dbpass@" />

That totally works. The resulting file in the build directory after a gradle installApp will look like this:

<property name="dbpass" value="${db.password}" />

as we want, and then, at runtime, the ${db.password} value will be de-referenced using the DB_PASSWORD environment variable.

That's fine, everything works, but why oh why!

When you see a target build time configured like this, new users might get confused about when the ${db.password} gets de-referenced—whether it happens at build time based on the build-time environment variable or at runtime. It’s very confusing, and even an experienced jPOS developer can get caught off guard. Don’t do it.

We mentioned CODE SMELL 1 and CODE SMELL 2, but there’s CODE SMELL 0!:

CODE SMELL 0 !

Configuration doesn’t belong in the same repository as the source code. That’s why we have a distnc (distribution, no-config) task in jPOS’ Gradle plugin that only ships the main jar and all dependencies in the lib directory, leaving the cfg and deploy directories to be fetched by other means. We tend to use GitOps for that, even before GitOps was a term; we did this in the old Subversion days. Having a deploy and cfg directory in your development repo is handy for development and testing, but we recommend keeping it separate from production.

jPOS 2.1.8 has been released

· One min read
Alejandro Revilla
jPOS project founder

jPOS 2.1.8 has been released. New development version is 2.1.9-SNAPSHOT and jPOS 3.0.0 (aka 'next').

See ChangeLog for details.

We will continue to maintain the 2.x.x series for a long while, so you can consider the 2.x.x series an LTS version, but all the fun and new features will happen in 3.x.x.

New Documentation

· One min read
Alejandro Revilla
jPOS project founder

I've recently embarked on an ambitious project to develop a new jPOS documentation, which is currently in its nascent stages. This tutorial is designed to evolve in tandem with ongoing jPOS development, and in time, it will replace existing documentation to become a dynamic, live tutorial and reference.

As it stands, the tutorial is in its early stages, but it already offers valuable insights for newcomers to jPOS, particularly with the setup primer and QBean information. Veterans of jPOS will find the sections on 'Environments' insightful, especially for applications in cloud deployments. This is also a prime opportunity to test and experience the jPOS Gradle Plugin in action.

Alongside the tutorial, we have a companion project on Github. It’s structured with distinct branches for each part of the tutorial, enabling you to access and execute the examples easily.

Since this is a work in progress and will be evolving alongside jPOS, your feedback is incredibly valuable. We encourage you to share your thoughts, suggestions, and insights, which will be instrumental in shaping this tutorial into a comprehensive and up-to-date resource for all things jPOS.

Here is the link.

Restricted Context

· 3 min read
Alejandro Revilla
jPOS project founder

The jPOS TransactionManager context, when it comes to traverse all participants, is quite promiscuous, as promiscuous as the global state of any application. It relies on the participants to do the right thing and don't step over each other, by mistake, or on purpose.

As applications grow and we move from a handful participants to several dozens, or even hundreds, it becomes very useful to put some constraints on what a participant can or can't do related to its Context.

On large jPOS applications, created by disparate teams, the TransactionManager configuration file is usually an excellent self-documenting starting point in order to figure out which transactions are supported, which participants are involved and how they are configured.

In addition, it is common for a participant’s implementation, before doing its job, to check for the presence in the Context of its needed input variables. CheckFields for example, needs a REQUEST object. SendResponse, needs a RESPONSE object.

To solve this, we've introduced a new required element, verified by the TransactionManager before calling the participant's prepare method, that looks like this:

<participant class="org.jpos.transaction.CheckFields">
<requires>REQUEST</requires>
</participant>

In this case, the TM checks for the presence of a REQUEST entry in the Context. If it’s not present, the transaction aborts right away (as if the participant had returned ABORTED).

This is just syntactic sugar, as the participant can check (and actually check) that, but there's a little twist.

The participant does not receive the whole context, it just receives a clone of it with JUST the entries described in the <require> element (which BTW, can be a comma separated list).

In addition to the require element, we support an optional element. i.e.:

<participant class="org.jpos.transaction.CheckFields">
<requires>REQUEST</requires>
<optional>TIMESTAMP, TIMEZONE</optional>
</participant>

You get the idea, if the optional TIMESTAMP and TIMEZONE entries are present, they would get copied to the mini Context used to call the participant.

A similar approach is used in order to selectively get the output of a participant. We use the provides element like this:

<participant class="org.jpos.transaction.CreateDSRequest">
<requires>REQUEST</requires>
<provides>RESPONSE</provides>
</participant>

The TransactionManager, who has access to both the original Context and the miniContext offered to the participant, merges back ONLY the entries listed in the provides method.

This is an experimental preview feature implemented in jPOS 3.0 (next). It needs to be battle tested in order to see if it makes sense to keep it, or it's just a useless idea. If we like it, and we continue supporting it, we may choose to enforce it on every participant. In a second phase we can define a TM-level promiscuousproperty so that if one chooses the old behavior, you need to set it true, either globally at the TM config level, or at the participant level, perhaps with a promiscuous="true" attribute.

jPOS 2.1.8 has been released

· One min read
Alejandro Revilla
jPOS project founder

jPOS 2.1.8 has been released. New development version is 2.1.9-SNAPSHOT and jPOS 3.0.0 (aka 'next').

See ChangeLog for details.

We will continue to maintain the 2.x.x series for a long while, so you can consider the 2.x.x series an LTS version, but all the fun and new features will happen in 3.x.x.

Q2 templates

· 3 min read
Alejandro Revilla
jPOS project founder

Typical jPOS applications have many similar servers, muxes, and channels. Sometimes hundreds of them.

In order to simplify its configuration Q2 now supports templates that can be launched from a template deployer, or programmatically using Q2's deployTemplate method.

A template is identical to a regular QBean XML descriptor, and can be placed anywhere in the filesystem, or classpath (including dynamic classpath) that has a special keyword called __PREFIX__ that are replaced at deploy time.

Q2's deployTemplate method has the following signature:

deployTemplate(String resource, String filename, String prefix);
  • resource is the resource location (e.g.: templates/channel.xml). If the resource starts with jar:, Q2 fetchs it from the classpath (e.g.: jar:META-INF/q2/templates/channel.xml).
  • filename is the QBean descriptor filename, e.g.: 10_acme_channel.xml
  • prefix is the Environment prefix (see below).

Imagine you have an environment (cfg/default.yml) file that looks like this:

    acme:
host: 192.168.1.1
port: 8000
emca:
host: 192.168.1.2
port: 8001

In a regular deployment, you could have two channel configurations like this:

    10_channel_acme.xml

<channel-adaptor name="acme-channel-adaptor" ...>
<channel class="${acme.channel}" packager=...>
<property name="host" value="${acme.host}" />
<property name="port" value="${acme.port}" />
...
...
</channel>
<in>acme-send</in>
<out>acme-receive</out>
</channel-adaptor>

10_channel_emca.xml

<channel-adaptor name="emca-channel-adaptor" ...>
<channel class="${emca.channel}" packager=...>
<property name="host" value="${emca.host}" />
<property name="port" value="${emca.port}" />
...
...
</channel>
<in>emca-send</in>
<out>emca-receive</out>
</channel-adaptor>

With templates, you can have a single template like this somewhere in your filesystem or classpath (say templates/channel.xml).

<channel-adaptor name="__PREFIX__-channel-adaptor" 
enabled="${__PREFIX__.enabled:false}"
class="org.jpos.q2.iso.ChannelAdaptor"
logger="${__PREFIX__.logger:Q2}">
<channel class="${__PREFIX__.channel}"
logger="${__PREFIX__.logger:Q2}"
packager="${__PREFIX__.packager:org.jpos.iso.packager.GenericPackager}">
<property name="host" value="${__PREFIX__.host}" />
<property name="port" value="${__PREFIX__.port}" />
<property name="header" value="${__PREFIX__.header}" />
<property name="connect-timeout" value="${__PREFIX__.timeout:5000}" />
<property name="packager-config" value="${__PREFIX__.packagerConfig}" />
</channel>
<in>__PREFIX__-send</in>
<out>__PREFIX__-receive</out>
<reconnect-delay>${__PREFIX__.reconnect:10000}</reconnect-delay>
<wait-for-workers-on-stop>${__PREFIX__.wait:yes}</wait-for-workers-on-stop>
</channel-adaptor>

and perhaps a mux.xml connected to it:

----
<mux name="__PREFIX__" class="org.jpos.q2.iso.QMUX"
enabled="${__PREFIX__.enabled:false}"
logger="${__PREFIX__.logger:Q2}" realm="__PREFIX__">
<key>${__PREFIX__.mux.key:41,11}</key>

<in>__PREFIX__-receive</in>
<out>__PREFIX__-send</out>

<ready>__PREFIX__-channel.ready</ready>
</mux>
----

Then you can deploy a QBean descriptor like this:

<templates>
<template resource="templates/channel.xml" descriptor-prefix="10_channel_">
acme,emca
</template>
<template resource="templates/mux.xml" descriptor-prefix="20_mux_">
acme,emca
</template>
</templates>

The special text __PREFIX__ is replaced by Q2 in each file using the prefix acme and emca, so the properties:

  <property name="host" value="${__PREFIX__.host}" />
<property name="port" value="${__PREFIX__.port}" />

gets converted to

  <property name="host" value="${acme.host}" />
<property name="port" value="${acme.port}" />

Very simple, it's just configuration sugar, but comes very handy in deployments with hundreds of services.

This is available in jPOS Next (series 3.x.x) starting in 3.0.0-SNAPSHOT availble in jPOS Maven repo, as well as Github next branch.

TM timeout and max-time

· 2 min read
Alejandro Revilla
jPOS project founder

The TransactionManager's participants provide easy ways to control per transaction timeouts. One can add a TIMESTAMP entry at the beginning of the transaction, and then have each participant control how it's going, so to speak. If the transaction has spent more than a threshold amount of time, it's better to abort the transaction right away because the client (i.e. POS device) probably has expired the transaction already and it's not waiting for our response anymore.

We can do that on each participant, or you can have a general-purpose timeout participant that you can place here and there.

But because this is a common use case, to encourage its use, in jPOS Next the TransactionManager honors a couple of new attributes at the participant level:

  • timeout
  • max-time

e.g.:

  ...
...
<participant class="org.jpos.jcard.CheckFields">
<property name="mandatory" value="PCODE,7,11,12,AMOUNT,PAN,41" />
</participant>
<participant class="org.jpos.jcard.SelectCurrency" />
<participant class="org.jpos.jcard.CreateTranLog" timeout="1000" max-time="5000">
<property name="capture-date" value="capture-date" />
<property name="checkpoint" value="create-tranlog" />
<property name="node" value="${jcard.node:99}" />
</participant>
...
...

In this example, three things can happen:

  • if, by the time the TM reaches the CreateTranLog participant, the transaction has taken more than 5000ms thus far, the participant is not called, and the transaction continues with the ABORTED route.
  • If the partitipant takes more than 1000ms to process, even if it has returned PREPARED, we coerce the response to ABORTED.
  • If max time, at the end of the participant is greater than 5000ms, we also coerce the response to ABORTED.

jPOS Next

· 3 min read
Alejandro Revilla
jPOS project founder

jPOS has evolved based on simple principles:

  • Latest version is always production ready
  • We don't rock the boat. All changes are backward compatible.

When Java started its new release cadence, we kept the baseline source compatibility to Java 8, while making sure jPOS was able to run on newer JDKs, but that prevented us to take advantage of new language features. There were features—like a long time waiting for multi-line Strings—that were nice to have, but one could still live without, but then there was one very interesting one for us, Project Loom (formerly fibers) that almost made it to Java 17, that we want to test at some large sites along with the TransactionManager.

Sealed classes, Records, Switch pattern matching, being prepared for JEP411 that will deprecate the Security Manager are all things we want to play with.

In addition, there are some changes we want to try in jPOS that would certainly rock the boat, related to audit logging (yes, we want to be agreegator/JSON/ELK friendly), using Strings instead of Integers in ISOMsg keys (so one doesn't have to create an arbitrary mapping on EMV related messages), Fluent APIs to create components for those that hate XML (funny, people find XML configs archaic but still love pom.xml), in-memory deployments, per-participant profiling on the TransationManager, etc.

In addition, we need to get rid of some baggage. The compat_1_5_2 module will remain available in the repo, as a reference, but it doesn't have to exist in the main tree. Some AWT/Swing based classes may go back to jPOS-EE as a module, and we may have a sibling module using JavaFX in order to see if it takes traction (JavaFX looks promising).

The CLI is up for a refactor. It's super useful to implement jPOS commands, but the API, initially very tied to JLine has proven to be kind of problematic once we added SSH support. We can do better on that side.

So... here is what we are planning to do:

  • jPOS 2.x.x series will become an LTS version with just security patches.
  • All the fun will happen in jPOS 3.x.x series, that currently has a baseline on Java 17 (but that may change if our Loom tests go well and we need it as a baseline).

For the time being, there's a next branch (already available in Github) that builds jPOS 3.0.0-SNAPSHOT.

The master branch currently has jPOS 2.1.8-SNAPSHOT but once we release 2.1.8, we'll switch next to become master (or take the opportunity to rename it to main) and use 2.1.8 as the forking point for an LTS v2 (or similar name) branch.

jpos-branch-plan

jPOS 2.1.7 has been released

· One min read
Alejandro Revilla
jPOS project founder

jPOS 2.1.7 has been released. New development version is 2.1.8-SNAPSHOT.

See ChangeLog for details.

Work has been started toward jPOS 3.0.0 targeting Java 17 as a baseline.

We will continue to maintain the 2.x.x series for a long while, so you can consider the 2.x.x series an LTS version, but all the fun and new features will happen in 3.x.x.