This is a personal story not related to jPOS, but it's somewhat related to payment networks and security, so I hope you enjoy it.

Back in the 80s here in Uruguay, when I was in my early 20s, credit cards started to become popular and merchants started to use CATs (credit authorization terminals) that used some mysterious protocol to talk to some servers in order to get authorizations.

I didn't have a card, but my partner in crime since age 7 -- my friend @dflc -- got one, I think it was a VISA.

We analyzed the card and of course, we were very interested to figure out what was written in that magnetic stripe, but we didn't have a reader. We probably tried with some tape recorder heads in order to get some audio, I don't remember, but I'm sure we had to try that.

One day, we called a store in the new mall in the city, Montevideo Shopping Center, for personal reasons (probably wanted to buy a present or something like that). Not 100% sure I was the one that placed the call, but I think I was. I'm very anxious, so I never asked our secretary to place the calls for me, if a number was busy I would dial 100 times in a minute until the call completes (and this was rotary dialing). If I recall correctly, the store was Pascualini, still popular these days. After busy, busy, busy, I got to hear some 'click click click click', followed by silence...

When you are into modems, and BBSs, there's not doubt what you do in a situation like that, you whistle! A simple short whistle starting at around 900Hz and going up to 1200~1300Hz is easy to whistle and you get V.21 and Bell 103 modems to start their connection establishing dance.

So I whistled (or my friend), and heard the modem, we knew it wasn't a FAX (no birps), we knew exactly what was that, that new tiny CAT thing, an Omron CAT 90 that we've started to see at some stores.

We knew exactly what was happening, that thing wasn't detecting that the line was free before blindly start dialing. Our eyes opened, we simultaneously smiled, it took us probably a few milliseconds to know what was next: Man in the middle!

We also saw a business opportunity (we were hungry): we knew we could build a little hardware to sense the DC voltage of a free versus busy line and sell it to the local acquirers (free line tone were not standard those rotary-dialing days).

As a first step, we planned for a proof of concept. We wanted to monitor a transaction, record it on tape, and let the real acquirer process the transaction. We were into BBSs and ran our own BBS those days. My friend had his home land line, plus 8 BBS lines in his bedroom (along side with a MicroVAX with two SCSI 500MB mirrored noisy disks spinning day and night), so we could use one line to dial the merchant, and another one to dial the acquirer. We had to do some war dialing and small social engineering to get the acquirer's listed phone numbers, lucky for us, numbers were in the phone book.

We played with phones since we were 8 or 10 years old, I remember I used to short-circuit the phone to break my mother's long calls when I needed it. We did phone patches for the ham radio stations, my friend @dflc used to develop his own telephone answering machine using discrete IC components (4011s and 4001s here and there) and a pair of cassette recorders, so the required hardware was ready in a couple days.

We needed a way to know when to initiate a call to the store exactly when the transaction was going to be initiated. There were no cell phones those days, but of course, we had VHF handhelds, actually a pair of Icom IC-02ATs. I used handhelds since high school, I thought they were the coolest thing to have and I still don't understand why ladies were not impressed by a guy with that kind of technology hanging in his belt, unbelievable...

[]

The distance between my friends' and the mall was small, just 600 meters

[]

The plan was easy: My friend (who owned the card) would go to the mall, buy something, send some signals (without talking, just a few push to talk pushes - for those in the know, that would be an A1 encoding) at the right time when the lady at the store was about to process the transaction. I'd be in our NOC (his bedroom) calling the merchant, the acquirer, hitting 'REC' on the recorder, and patching both lines with our little hardware (also monitoring with headphones).

We did a test VHF connection and although those handheld transceivers could be used to cover 80+ kilometers with good conditions, the mall was a Faraday cage, I didn't hear him. So we needed a plan B. My friend's brother got a mobile VHF in his car, plenty of power (50W). So we called him (via radio) and luckily he was close to the area. We explained the mission, although no questions were asked, he would take us seriously. He parked the car close to the mall (so he could listen the short transmission from inside the mall) and QSP to me. (QSP, Q2 aka QSP version 2, rings a bell?). FTW, QSP is the Q-signal code for "relay message".

So we did the transaction, everything worked on the first try, @dflc bought himself a leather wallet or a belt, can't remember, knowing him, I'm sure he still have it as a trophy and a way to remember that fun hacking day. The transaction was properly approved by the Visa acquirer (who BTW, now runs jPOS), we were just men in the middle.

We've got the transaction and I remember we analyzed it in several ways, replayed it against different modems, etc.

On the business side, we had meetings with the local card acquirers where we explained their vulnerability and offered a solution. Of course, they didn't like the fact that we, young suspicious "hackers/crackers" were telling them what to do so they did nothing.

We kept our grin for a good while, it was a nice, albeit pretty simple, hack.

Every single week, for the last 14 years I have discussions with developers, CTOs and CIOs about channel timeouts.

The discussions usually start by a customer requirement asking us to keep established socket connections forever.

They say "We want the socket to stay always connected, forever. We don't want to see disconnects. Our systems are very reliable, our remote endpoint partners are very reliable, we don't want a timeout".

So I usually start with the Fallacies of distributed computing but I'm never lucky. I try to explain that I don't want to die, but it just so happens that I will certainly die, sooner or later. It's life.

Disconnections happen, networking problems happen all the time, router and firewall reboots, and the most evil situation, a paranoid firewall administrator configuring very tight timeouts.

When jPOS is the client, and the channel is idle for a long period of time, having no timeout is actually not a big deal. Imagine a situation where the channel is connected for say 5 minutes, but our paranoid FW administrator had set a timeout of 3 minutes to disconnect the session. While jPOS believes we are connected, we are actually not connected, so when a real transaction arrives, and we try to send it, we find out we are no longer connected. That will raise an exception, we'll reconnect, and we'll send the message (a few seconds later). So the problem is just a delay that may put us out of the SLA for this particular transaction, but it's still not a big deal, the system will recover nicely.

But when jPOS is the server, and we don't have a timeout, the client will establish a new connection, but the old one will remain connected forever. A few hours/days later, these connection will accumulate and we'll hit the maxSessions of the QServer configuration (see the Programmer's Guide section 8.4). Only way to recover is to restart that particular QServer, something that needs to be done manually.

You can set SO_KEEPALIVE at the channel level in order to detect these broken connections, and in order to prevent some firewalls from disconnecting your session, but the KEEPALIVE time is OS dependent.

Our recommendation is to send network management messages from time to time (i.e. every 5 minutes) and have a reasonable timeout of say 6 minutes.

There's another situation where you want a timeout. Imagine an ideal network (I call it 'Disney LAN') where the connection remains ESTABLISHED from a TCP/IP standpoint, but the remote host's application is dead and is not answering to your replies. You can of course detect that at the application level (i.e. MUX) and proactively initiate a reconnection, but if that logic fails (or you never implemented it), a reasonable timeout will recover automatically from the situation. The remote host doesn't reply, the call to channel receive time us out out, we reconnect, and with a little bit of luck, we get to connect to a new session that actually works.

First a disclaimer: I know excellent auditors, starting with my friend Dave from the Payments Systems Blog, but I also know really retarded ones, and here is a little story of a system I built some 8 or 10 years ago that would have been resilient to the HeartBleed bug, but of course, the auditor couldn't understand it, and it had the word MD5 which puts them to cry like tiny little girls, so we have to "improve" it to make it less secure.

I've been into amateur packet radio and BBS systems in the 80s where monitoring the air, or a serial line was easy, so things like one time passwords and two factor authentication have been always within my area of interest. When it came to provide an internal user interface to jPOS, and I had to design a login form, I wanted to protect the user's passwords against an operator with access to the system; I wanted people to be able to use a password like TheBossIsAnIdiot if they wanted, making it difficult for the programmer/operator on the server side to see it.

So the solution was easy. The server would generate a nonce and send it to the client, the client would use that nonce, some other data (like the session id) and the password, and send an MD5 of all that to the server.

I wasn't and I'm not a JS expert, and we didn't have things like jQuery or Angular those days, but I wrote this little piece of code that implemented the login form:

    function doHash(frm) {
        var username = frm.username.value;
        var password = frm.password.value;
    
        if (username.length < 3 || password.length < 3) {
            alert ("Invalid Username and/or Password.");
            return false;
        }
        var hash     = frm.hash.value;
        var seed     = readCookie ("JSESSIONID") + hash;
        var pass     = hex_md5 (username + password);
    
        frm.password.readOnly = true;
        frm.password.value = hex_md5 (seed + pass);
    }

The server would do the same computation in order to verify the login.

I wasn't comfortable with the solution, because somehow the initial password was either entered by the operator, or sent via email, so I forced a password change in the first login, and the password change would just send an XOR of the existing password hash so that the server could apply the same XOR and upgrade the password to the latest version.

But here comes the auditor, with a bucket in his head, and reports the process as insecure (with strong copy/pasted wording to scare management) for the following reasons:

1. On password change, the complexity of the password (you know, password length, use lower and upper and all that crap) is validated in client side, not in the server side. And there goes a rant that says "Passwords should be at least XXX characters, and have lower case and upper case letters in order to be secure, yada yada yada", so the manager would look at me like "We trusted in you... look what you did to us, our passwords can be less than XXX characters if the user hacks the client side code!").

2. It has the word MD5 and we just heard that MD5 is broken (remember this was 2005~2006), and there goes the rant about how MD5 was recently cracked, and again, the manager would give you that look, like saying, you're a lost case, I've been always scared of open source and the freetards around it.

I think the tradeoff between having one user hacking the JS to force himself a weak password, compared to protecting all users from easy eavesdropping is a good one. I also think that sending an MD5 over the wire is better than sending a clear password (although there's of course SSL involved, I'm talking about 'clear' from an application perspective). It has the side benefit of staying secure while in memory on the server side.

Auditors are obsessed with the things they were told to look after, SQL injection, XSS, or things they test with automated tools. That's fine and welcome (you don't need an auditor for that, BTW, but it's good to have more eyes on the problem). But I've never seen an auditor testing to SQL inject say a field 35/45 in an ISO8583 message, something anyone can do forging the track2 of a card and going to a shop around the corner. Take what they say with a grain of salt, and remember not all, but most of them, are just talkers.

I'm playing with the idea of making the client perform a really large number of iterations on the hash, to slow it down (kind of a client side bcrypt) without requiring too much CPU on the server, and then have the server do some more rounds, perhaps with a bcrypt final pass. I'm planning to send some timing information to the server too, in order to alert on client hardware/software changes (how long did it take you to run 100K hashes?). We'll have to figure out how to explain this in our next audit..

In jPOS 1.9.7 cce6a27 we've added a new transient trace flag to the Context that can be very useful during development.

Those of you using the TransactionManager with a large number of participants know that sometimes it becomes difficult to know who placed what in the Context.

You get to see a Context with many entries (REQUEST, RESPONSE, IRC, SOURCE, TRANLOG, TIMESTAMP, AMOUNT, PAN, ADDITIONAL_AMOUNT, etc.) but pin pointing where a given value is place gets difficult.

If the Context new trace boolean is set to true (something you can do via a configuration property in one of the initial participants such as PrepareContext or even closer to the incoming message, in the ISORequestListener when you create the Context), the Debug output would look like this:

Although it may look verbose, this could be very useful while coding, it helps you spot problems and assist on debugging.

I just found one issue in the jCard system while writing this blog post, look at this, we set the ACCOUNT in SelectAccount

ACCOUNT='org.jpos.gl.FinalAccount@7f7a1bec[id=28,code=22.0000000002]' [org.jpos.jcard.SelectAccount.prepare(SelectAccount.java:49)] [1.0/75.6]

then we set it again in Authorization.

ACCOUNT='org.jpos.gl.FinalAccount@7f7a1bec[id=28,code=22.0000000002]' [org.jpos.jcard.Authorization.prepare(Authorization.java:110)] [0.1/108.2]

Not a big deal, it's the same account, but worth checking why we are doing that.

The jPOS QMUX service uses the Space (usually the default global space) in order to communicate with other components such as the ChannelAdaptor or QServer using its in and out queues. But in addition, it implements the MUX interface by storing selected parts of a request message (known as the QMUX key), as shown in the picture below:

In high traffic systems, with many QMUXes, every thread waiting for a response would wake up, albeit for a tiny little while, when something happens in the space. This small patch done in 1.9.7 keep using the global space for the QMUX in and out queues, but uses an internal Space (currently a TSpace) to perform the key-matching dance.

The change should be transparent for most users, but we've seen some implementations out there that dangerously peek and poke our entries in the Space, and actually this patch, in addition to improve performance, intend to discourage such use in the future (by not exposing the internal space to other components). But anyway, for backward compatibility, we honor a new property reuse-space that if set to true, would revert to the old implementation, using the global space.

You might have heard a thousand times, push is good, IoC is good. pull is bad, and I have to agree.

jPOS components get their configurations pushed by the Q2 container when they implement the Configurable interface.

But if you're used to jPOS configurations, which can be filtered at build time by the Gradle build based on the desired target profile, or can be decorated by means of @vsalaman's contributed decorator, you may find yourself reinventing the wheel and figuring out how to get some Configuration object into a non jPOS component (such as a servlet or any other non jPOSsy code).

To solve that in a standard way, we've created QConfig. QConfig is a minimalistic QBean that just register its own Configuration object into the NameRegistrar (with a "config." prefix). So for example, you can deploy something like this:

The word config has been registered in QFactory.properties so the <config> element above is equivalent to:

So non jPOS running inside Q2 can get a reference to 'config' configuration by calling:

Configuration cfg = QConfig.getConfiguration("myconfigname");

While we were at it, we added the ability to merge configuration objects in other QBeans; There many ways to achieve the same without using this technique, for example, you can use <property file="xxx" /> in different QBeans to pull the same config, or you can use XML entities for that, but, because we can, we just offer this additional way to do it, which is quite simple.

Any QBean descriptor now accepts an optional attribute called merge-configuration that accepts a list of QConfig configurations and merges them on-the-fly at QBean configuration time. Here is a simple example:

deploy/00_config.xml

deploy/01_config.xml

deploy/90_script.xml

Because this merge-configuration handling is honored by QFactory, used by other components such as the TransactionManager to instantiate its participants, you can use it in TM participants as well (i.e. to pull reused configuration, such as result codes and the like).

Now that 1.9.4 has been released, we moved the development version to 1.9.5-SNAPSHOT (nightly builds available in jPOS Maven Repository).

We've added DUKPT support to JCESecurityModule. There's no documentation yet, but for a an easy to read example, you can take a look at the DUKPTTest class.

jPOS 1.9.4 has been released and it includes the following changes, most notably OSGi support.

• Added len,description constructor to IF_NOP
• Added IFB_LLHEX (can be used to deal with encrypted track2s)
• Added HexNibblesPrefixer (required by IFB_LLHEX)
• Added OSGi support
• Added 'qnode' (OSGi testbed)
• DirPoll now supports file compression
• Profiler can be reenabled
• TransactionManager PAUSED transactions reuse profiler to provide combined elapsed times
• Added org.jpos.iso.GenericSSLSocketFactory
• jPOS-105 QServer should unregister as a space listener
• jPOS-106 ChannelAdaptor reconnect flag uses non serializable Object
• jPOS-108 FSDMsg consuming input stream
• DirPoll.scan is now protected
• MUX interface now extends ISOSource
• QMUX.send now checks isConnected()
• DirPoll now accepts a priority.regex boolean property (73c2f84)
• jPOS-110 QMUX major start-up issue (1526dab)
• DirPoll Retry when archive.timestamp is set to true (pull/33)
• Generate optional app specific version info 02f739a

See full ChangeLog.

1.9.4 is available in Maven Central.

What API designers could learn from the payments industry​

We all know the Fallacies of Distributed Computing:

1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.

I think there's a 9th: REST API designers know the meaning of the word Fallacy.

So to clarify and as a public service, we need to start talking about distributed computing facts instead:

1. The network IS NOT reliable.
2. Latency IS NOT zero.
3. Bandwidth is limited.
4. The network is not secure (you should know that already)
5. Topology does change, and at the worst possible time.
6. The administrator is dead.
7. Moving bits around the net has a cost.
8. The network heterogeneous.

Interesting enough, a very old protocol, ISO-8583, designed in the 80s to support slow 300 and 1200 bps dialup links is extremely aware of these facts and work around these problems with a very simple and asynchronous message flow.

Take a look any popular REST payments API, you usually call a method to authorize a transaction and you pass parameters like this:

• Card and Expiration (or a token)
• Amount, perhaps currency
• A description

Lovely, simple, and wrong!

Using that popular design, you POST the transaction and pray to get a response from the server. If you get a response, either a 200, 201, or even a 500 from the server, everything is alright, but if the request times out, or you go down, or the servers goes down, or the ISP is reconfiguring a router, you can't really tell what happened. If you're lucky and the server didn't receive your request, then that's fine, you can retry it, but if the server did receive your request, and authorized against its upper level acquirer, then you'll have an angry cardholder with a hold in its account, and perhaps even a debit (because I don't see many payment gateways accepting reconciliation messages or settlement files).

In ISO-8583, when you send an authorization request or a financial request and you don't get a response, you queue a reversal in a persistent store and forward (SAF) queue. So the next time you contact the server, before sending a new transaction, you send that reversal. If you receive a response from the server, but the response comes late and you have already timed out, you also send a 'late-response' reversal to the server.

In the same way, when you post a transaction that already took place (i.e. adjusting a previously approved transaction for a different amount, something that happens all the time at restaurants that support tips, or gas pump transactions where you approve for $100 but then complete for $20), and you don't get a response, you send a retransmission of the same transaction, as many times as necessary in order to deal with transient network problems.

In order to support reversals and retransmissions, you need a transaction unique identifier. Different networks use different transaction identifiers, either the Terminal ID + a Serial Trace Audit Number, or a Retrieval Reference Number, or in ISO-8583 v2003 the new Transaction life cycle identification data, the client generates a unique identifier so that when you send a follow-up transaction, you can send the server a reference to the original one.

I believe all payment APIs out there (including those super very cool ones) should consider adding three things:

A new parameter, RRN​

Client code could generate a UUID and use it as the RRN (Retrieval Reference Number)

Support reversals (DELETE)​

Could be as simple as adding a new verb, DELETE, where the only parameter can be -- along with authentication data -- the RRN

Support for retransmissions (PUT)​

If your initial transaction was a POST, I propose that you also accept a PUT, with the same parameters. On the server side the difference between the POST and the PUT operations would be just an extra step to make sure that you didn't process the original POST, and return the original auth code if that was the case.

Of course, if you're designing a new super cool minimalistic REST API you probably don't listen to people with grey hair, but just in case, my 2c :)

@apr

For those not regularly checking the [jPOS main site], please note there's a new Learn tab in the main site with a direct link to download the new and free jPOS Programmer's guide draft.

While it's still work in progress, it provides useful information related to jPOS 1.9.x series that complement the standard for-sale guide.

Feedback is of course very Welcome!