JVM Memory Options
/2019-08-28/
Q: In the /bin/q2 script of jPOS, -Xmx1G is set. Will it be ok to set it to -Xmx16G or higher for a Windows Server of 32RAM? Advice and directions will be highly appreciated. Thanks great community.
A:
It’s totally OK to increase, but if you increase it, watch out your GC configuration. This blog post is old, and the switches have changed in more recent JVMs, but it’s useful to shed some light on the kind of things that you want to watch out in order to avoid and monitor stop-the-world GC cycles.
QServer as client
/2019-08-29/
Q: I need to send a message out the door to the TCP/Client that is connected to this server.
A:
Despite the fact that QServer is a server from a TCP/IP standpoint, it can be a client at the application level.
sequenceDiagram
participant Acquirer
participant QServer
Acquirer->>QServer: Acquirer initiates TCP/IP connection
QServer->>Acquirer: ISO-8583 request
Acquirer->>QServer: sends ISO-8583 response
This can be easily achieved by adding an <in> and <out> queue to the QServer configuration, in the same way you’d use it in a ChannelAdaptor, i.e.:
<server class="org.jpos.q2.iso.QServer" logger="Q2" name="xml-server">
<attr name="port" type="java.lang.Integer">8000</attr>
<channel class="org.jpos.iso.channel.XMLChannel"
logger="Q2" packager="org.jpos.iso.packager.XMLPackager">
<property name="timeout" value="3600000" />
</channel>
<in>remote-send</in>
<out>remote-receive</out>
</server>
Those in/out queues can be connected to a QMUX, i.e.:
<mux class="org.jpos.q2.iso.QMUX" logger="Q2" name="isobridge">
<in>remote-receive</in>
<out>remote-send</out>
</mux>
(Quick reminder: in/out queues are seen from the component’s perspective, when you want to “send” something to say a ChannelAdapter, you push the ISOMsg to its in queue. Whatever the ChannelAdaptor receives on its TCP/IP ISO-8583 side, it will place it in its out queue - same goes for the QServer, and QMUX).
If in addition to sending messages to the remote TCP/IP client you want to receive messages coming from it, you can use an ISORequestListener. The typical use has the ISORequestListeners in the QServer, but in this configuration, you need to move them to the QMUX.
So the QServer would just use the in/out queues to communicate with other components, and the QMUX would take care of handling outgoing as well as incoming messages.
graph LR Client --> QServer QServer --> QMUX QMUX --> QServer QMUX --> ISORequestListener ISORequestListener --> QServer
Now, this is important: when client code uses the QMUX to send out a request to the remote host, by placing it in the QServer’s in queue, QServer will send it out through the last connected ISOChannel (there’s a handy QServer.getLastConnectedISOChannel() method for that).
When QServer listens on a given port, many clients can connect:
graph LR Client1 --> QServer Client2 --> QServer Client3 --> QServer QServer --> QMUX
If the connections come in order (client 1, client 2, client 3), if you call mux.request, QServer will send it through the socket connection to client 3.
If this is your intended use-case, you can stop here. But there are some other use cases, for example, when you want to initiate a network management message upon connection:
sequenceDiagram
participant Acquirer
participant QServer
Acquirer->>QServer: Acquirer initiates TCP/IP connection
QServer->>Acquirer: 0800
Acquirer->>QServer: 0810
Acquirer->>QServer: 0200
QServer->>Acquirer: 0210
In this situation, one typically want to send a 0800 message on ALL connections to QServer.
In addition to the request-listener that one can add to the QServer configuration (or QMUX), there’s also a connection-listener that gets called once QServer accepts a connection and can be used to trigger a 0800, or to register the new channel with a custom connection manager.
The configuration would look like this:
<server class="org.jpos.q2.iso.QServer" logger="Q2" name="xml-server-8000" realm="xml-server-8000">
<attr name="port" type="java.lang.Integer">8000</attr>
<channel class="org.jpos.iso.channel.XMLChannel"
packager="org.jpos.iso.packager.XMLPackager"
type="server"
logger="Q2"
realm="xml-server-8000">
<property name="timeout" value="180000"/>
</channel>
<request-listener class="org.jpos.iso.IncomingListener" logger="Q2" realm="incoming-request-listener">
<property name="queue" value="TXNMGR" />
<property name="ctx.DESTINATION" value="jPOS-AUTORESPONDER" />
</request-listener>
<connection-listener class="com.my.custom.Logon" />
</server>
The Logon class can look like this:
import org.jpos.iso.*;
import org.jpos.util.Log;
import java.util.Date;
import java.util.EventObject;
import java.util.concurrent.atomic.AtomicLong;
public class Logon extends Log implements ISOServerEventListener {
AtomicLong stan = new AtomicLong();
@Override
public void handleISOServerEvent(EventObject event) {
if (event instanceof ISOServerAcceptEvent) {
ISOServer server = (ISOServer) event.getSource();
ISOChannel channel = (ISOChannel) event.getISOChannel();
try {
channel.send(createMsg("001"));
} catch (Exception e) {
error(e);
}
}
}
private ISOMsg createMsg (String msgType) throws ISOException {
ISOMsg m = new ISOMsg ("0800");
m.set (7, ISODate.getDateTime (new Date()));
m.set (11, ISOUtil.zeropad (Long.toString (stan.incrementAndGet()), 6));
m.set (70, msgType);
return m;
}
}
This simple example blindly sends a 0800 message on the newly created channel, but users may want to register the channel internally so that network management messages can be regularly sent.
WARNING: If you keep a reference to channel in your ISOServerEventListener callback, we suggest to use a WeakReference so that the channel doesn’t create a memory leak once disconnected.
In addition to registering channels at connection time using a connection listener, QServer has a handy getISOChannelNames() method that returns a space-separated list of channel names. That list can be split and iterated using the getISOChannel(String) method to get a reference to a connected channel, and that channel can be used to regularly fire network management messages.
Upgrade old jPOS project
/2019-09-09/
Q: I have a project with jPOS that is build with ant, It's a very antique project and I want to update it to a newer version using gradle. Is there a standard procedure or a tutorial to follow in this case?
A: You may want to take a look at the jPOS-template. We use it as a reference whenever we have to create a new jPOS project. You may also want to check jPOS tutorials - in particular, the second one guides you through the creation of a simple jPOS project.
ProtectedLogListener
/2019-09-18/
Q: How can I remove sensitive information on the logs, and what is the proper way to handle logging process in jPOS?
A:
You can add the ProtectedLogListener to your logger configuration. If you’re using the default Q2 logger, configured in deploy/00_logger.xml, you could add something like this:
<logger name="Q2" class="org.jpos.q2.qbean.LoggerAdaptor">
<log-listener class="org.jpos.util.ProtectedLogListener">
<property name="protect" value="2 14 34 35 45 111.2 111.14 111.35 111.45" />
<property name="wipe" value="52 55" />
</log-listener>
<log-listener class="org.jpos.util.SimpleLogListener" />
<log-listener class="org.jpos.util.DailyLogListener">
<property name="prefix" value="log/q2" />
<property name="date-format" value="-yyyy-MM-dd-HH"/>
</log-listener>
</logger>
By adding the ProtectedLogListener before other log listeners, the fields configured in the protect will be properly masked, and those configured as wipe will get wiped.
The message would look something like this:
<isomsg direction="incoming">
<field id="0" value="0200"/>
<field id="2" value="411111______1111"/>
<field id="3" value="003000"/>
<field id="4" value="000000000100"/>
<field id="7" value="0903190443"/>
<field id="11" value="253632"/>
<field id="12" value="044359"/>
<field id="13" value="0903"/>
<field id="14" value="____"/>
<field id="18" value="5533"/>
<field id="19" value="036"/>
<field id="22" value="012"/>
<field id="25" value="00"/>
<field id="41" value="00000001"/>
<field id="42" value="000000000000001"/>
</isomsg>
Warning related to ISODate pattern YYYY
_/2019-12-31 taken from jPOS' Slack/
_from @Gandhi
Hi there!
We found ourselves in a weird situation with ISODate.formatDate.
After 2019-12-29 (this weekend), some problems with ICC validation rised from nothing (no changes made in the prod environment).
It happens that dates formatted with pattern YYMMdd were presented as 201229.
Don't know if you were aware of that kind of stuff or if it's documented somewhere, but it would easily corrected using another pattern. (all code comments uses "YY" as example)
It would help too, if ISODate has in it's comments and docs what patterns should be better to be used. I know that's more a missuse case, but if our mistakes can help someone to prevent theirs... it's a good thing right?Here's a code as example/demonstration:
import java.util.Date;
import org.jpos.iso.ISODate;
public class TesteISODate {
public static void main(String[] args) {
String[] ptt = {"YYMMdd", "yyMMdd"};
String[] strdate = {"20191228000000", "20191229000000"};
for (String dtStr : strdate) {
for (String pttStr : ptt) {
System.out.println(TesteISODate.getOutStr(pttStr, dtStr));
}
}
}
private static String getOutStr(String ptt, String strdate) {
Date dtin = ISODate.parseISODate(strdate);
return "Using pattern: " + ptt + " and Date for " + strdate + ": "
+ ISODate.formatDate(dtin, ptt);
}
}
Then Relevant tweet provided by @Soares
PSA: THIS IS THE SEASON TO CHECK YOUR FORMATTERS, PEOPLE TIL that Java's DateTimeFormatter pattern "YYYY" gives you the week-based-year, (by default, ISO-8601 standard) the year of the Thursday of that week.12/29/2019 formats to 2019 12/30/2019 formats to 2020
QBean File Names
00_logger.xml is the only special file that gets read ahead of anything else;
then you can use whatever numbering scheme you want. We tend to use:
10_*for channels20_*for MUXes30_*for TM50_*for Servers90_*for scripts99_sysmon.xmlfor the System Monitor
But that’s totally arbitrary.
Logon Manager
Many hosts require a sign-on network management message once a connection is established, regular echo messages during the lifetime of the connection, and a sign-off message once the connection is going down.
You can go low level and implement it as part of your channel implementation. Alternative, you can adopt the jPOS way of using a more decoupled Space based approach.
Here is how such approach would look like:
When a ChannelAdaptor connects to a remote host, it places a ready-indicator in the Space. That's the same ready-indicator used by the MUX in order to tell if the channel is connected or not. So if you have a channel called foo, once connected, it will place an object (actually an instance of java.util.Date, but don't rely on it as it could be something else in the future) called foo.ready in the space.
That means that you can have a service waiting for foo.ready to be present, and firing a 0800 message right after.
The configuration would look like this (i.e. deploy/30_logonmanager.xml)
<logonmanager class="org.jpos.jpts.LogonManager" logger="Q2">
<property name="persistent.space" value="je:logonmgr" />
<property name="timeout" value="15000" />
<property name="echo.interval" value="60000" />
<property name="mux" value="mymux" />
</logonmanager>
package org.jpos.jpts;
import java.util.Date;
import java.io.OutputStreamWriter;
import java.io.ByteArrayOutputStream;
import org.jdom2.Element;
import org.jdom2.output.XMLOutputter;
import org.jpos.q2.QBeanSupport;
import org.jpos.core.Configuration;
import org.jpos.q2.iso.QMUX;
import org.jpos.space.Space;
import org.jpos.space.SpaceUtil;
import org.jpos.space.SpaceFactory;
import org.jpos.core.ConfigurationException;
import org.jpos.iso.ISOUtil;
import org.jpos.iso.ISOMsg;
import org.jpos.iso.packager.XMLPackager;
import org.jpos.iso.MUX;
import org.jpos.iso.ISODate;
import org.jpos.iso.ISOException;
import org.jpos.util.NameRegistrar;
@SuppressWarnings("unused unchecked")
public class LogonManager extends QBeanSupport {
private Space sp;
private Space psp;
private long timeout;
private long echoInterval;
private long logonInterval;
private long initialDelay;
private ISOMsg logonMsg;
private ISOMsg logoffMsg;
private ISOMsg echoMsg;
private static final String TRACE = "JC_TRACE";
private static final String LOGON = "JC_LOGON.";
private static final String ECHO = "JC_ECHO.";
public void initService () throws ConfigurationException {
Configuration cfg = getConfiguration();
sp = SpaceFactory.getSpace(cfg.get("space", ""));
psp = SpaceFactory.getSpace(cfg.get("persistent-space", ""));
timeout = cfg.getLong("timeout", 30000);
echoInterval = cfg.getLong("echo-interval", 30000);
logonInterval = cfg.getLong("logon-interval", 86400000L);
initialDelay = cfg.getLong("initial-delay", 1000L);
Element config = getPersist();
logonMsg = getMsg("logon", config);
logoffMsg = getMsg("logoff", config);
echoMsg = getMsg("echo", config);
}
public void startService () {
for (String mux : cfg.getAll("mux")) {
new Thread(new Runner(mux), getName() + "-" + mux).start();
}
}
public class Runner implements Runnable {
String name;
MUX mux;
String readyKey;
public Runner(String name) {
this.name = name;
try {
mux = NameRegistrar.get("mux." + name);
String[] readyIndicators = ((QMUX) mux).getReadyIndicatorNames();
if (readyIndicators != null && readyIndicators.length > 0)
readyKey = readyIndicators[0];
else
getLog().error("Ready indicator for MUX " + name + " not configured.");
}
catch (NameRegistrar.NotFoundException e) {
getLog().warn(e);
}
}
public void run () {
while (running() && readyKey != null) {
Object sessionId = sp.rd(readyKey, 60000);
if (sessionId == null) {
getLog().info("Channel " + readyKey + " not ready");
continue;
}
try {
if (!sessionId.equals(sp.rdp(LOGON + readyKey))) {
doLogon(sessionId);
Thread.sleep(initialDelay);
}
else if (sp.rdp(ECHO + readyKey) == null) {
doEcho();
}
}
catch (Throwable t) {
getLog().warn(t);
}
ISOUtil.sleep(1000);
}
stopService();
}
public void stopService() {
try {
doLogoff();
}
catch (Throwable t) {
getLog().warn(t);
}
}
private void doLogon(Object sessionId) throws ISOException {
ISOMsg resp = mux.request(createMsg("001", logonMsg), timeout);
if (resp != null && "0000".equals(resp.getString(39))) {
SpaceUtil.wipe(sp, LOGON + readyKey);
sp.out(LOGON + readyKey, sessionId, logonInterval);
getLog().info("Logon successful (session ID " + sessionId.toString() + ")");
}
}
private void doLogoff () throws ISOException {
SpaceUtil.wipe (sp, LOGON+readyKey);
mux.request(createMsg("301", logoffMsg), 1000); // do not logoff
}
private void doEcho () throws ISOException {
ISOMsg resp = mux.request(createMsg("301", echoMsg), timeout);
if (resp != null) {
sp.out(ECHO + readyKey, new Object(), echoInterval);
}
}
private ISOMsg createMsg (String msgType, ISOMsg merge) throws ISOException {
long traceNumber = SpaceUtil.nextLong(psp, TRACE) % 1000000;
ISOMsg m = new ISOMsg ("2800");
m.set(7, ISODate.getDateTime(new Date()));
m.set(11, ISOUtil.zeropad(Long.toString(traceNumber), 6));
m.set(70, msgType);
if (merge != null)
m.merge (merge);
return m;
}
}
private ISOMsg getMsg(String name, Element config) throws ConfigurationException {
ISOMsg m = new ISOMsg();
Element e = config.getChild(name);
if (e != null)
e = e.getChild("isomsg");
if (e != null) {
try {
XMLPackager p = new XMLPackager();
p.setLogger(getLog().getLogger(), getLog().getRealm()
+ "-config-" + name);
m.setPackager(p);
ByteArrayOutputStream os = new ByteArrayOutputStream();
OutputStreamWriter writer = new OutputStreamWriter(os);
XMLOutputter out = new XMLOutputter();
out.output(e, writer);
writer.close();
m.unpack(os.toByteArray());
}
catch (Exception ex) {
throw new ConfigurationException(ex);
}
}
return m;
}
}
BONUS 0: multiple MUXes
The above example works with a single configuration file for multiple MUXes. You just need to repeat the mux property in the configuration and the QBean will handle as many connections as you want.
BONUS 1: message template
That private getMsg(...) method is used to tweak the different type of messages (logon, logoff, echo) so that you can have a configuration like this:
<logonmanager class="org.jpos.jpts.LogonManager" logger="Q2">
<property name="persistent.space" value="je:logonmgr" />
<property name="timeout" value="15000" />
<property name="echo.interval" value="60000" />
<property name="mux" value="mymux-0" />
<property name="mux" value="mymux-1" />
<property name="mux" value="mymux-2" />
<logon>
<isomsg>
<field id="70" value="001" />
</isomsg>
</logon>
<echo>
<isomsg>
<field id="70" value="301" />
</isomsg>
</echo>
<logoff>
<isomsg>
<field id="70" value="002" />
</isomsg>
</logoff>
</logonmanager>
You can of course override other fields, which sometimes require arbitrary data (such as an acquirer ID in field 32).