miniGL Layers

/by apr/ I´ve added a new feature to miniGL: Layers.

This new feature is designed to support multiple balances for a single account, such as available versus accounting balance.

We could use some flags for unsettled transactions, or pre-auths waiting for completion, but with this new layers support
everything gets more simple, reusable and IMO elegant. Imagine a simple ATM withdrawal transaction:

Debit Credit
Cardholder’s checking account
(We take money out of the cardholder’s account)
ATM Network
(We owe the money to the network now)

(both liabilities as seen from an issuer perspective, fees not included here for simplicity)

Sounds very nice, but we have a problem, this transaction is still unsettled until we get a batch close or completion transaction, so if we don´t have this new layers feature, we would end up having either some kind of flag or duplicating accounts, i.e:

  • Cardholder´s checking account
  • Cardholder´s checking account (pending)
  • ATM Network account
  • ATM Network account (pending)

With our new layered approach, we post real transactions to layer zero (ground) and pending transactions to say layer one. When the transaction gets settled at the end of the day, we just reverse the effect of the transaction in layer one (blue) and post it to layer zero (black), e.g, when we get a pre-auth, we could post:

Pre Authorization

Debit Credit Layer
Cardholder’s checking account
(We take money out of the cardholder’s account)
20.00 1
ATM Network
(We owe the money to the network now)
20.00 1

(In this example, ‘1’ would be the ‘pending’ layer).


Debit Credit Layer
Cardholder’s checking account -20.00 1
ATM Network -20.00 1
Cardholder’s checking account 20.00 0
ATM Network 20.00 0

(Now it´s confirmed, we “ground” it)

I´ve modified the existing rule implementations in order to support this new layers feature, e.g. the DoubleEntry rule (that verifies that debits equals credits in the reporting currency for a given GLTransaction) now accepts a list of layers to check, so we can be very strict in some layers (such as the ground layer aka the real deal) while more relaxed in other ones. minigl.dtd has been modified in order to support a new <layers> element in rule definitions.

Some other uses for this include data-entry application where a junior clerk could post transactions to a given layer for later verification, then someone else with enough karma to authorize the transaction would “ground” it. We can use this feature to track “budgets”, “cost centers” or just to ease some rule implementations (e.g.: we can have minimum balances layer, a max balances layer, and so on).

The layer number is arbitrary to a given miniGL application. On multi-currency systems, we use the currency ISO number as the accounting layer for a given currency.

When you ask for a balance for a given account at a given journal, miniGL gives you the ability to get the balance for a set of layers. This comes very handy when computing accounting versus available balances in ATM and POS transactions. You ask getBalance(journal, account, 0) and you get the accounting balance. You ask for getBalance(journal, account, 1) and you get the pending transactions (still unsettled). But if you ask getBalance(journal, account, 0,1) miniGL combines both layers to give you the available balance. We also use a third layer for overdrafts (used in credit card applications), say layer ‘2’, so on certain transactions that would allow an overdraft (i.e. POS purchases), we could check getBalance(journal, account, 0,1,2).

I just hope the guy in the mail stamp up there is not twisting in his crave … :)

TransactionManager RETRY

/by apr/ For some time I have been playing with the idea of supporting a PAUSE action at prepare time in the jPOS transaction manager (think continuations), but it has some side effects related to transaction recovery, so it is still in the works. But I´ve added an easy to use and handy RETRY action that you can use while implementing your participants. If you return RETRY at prepare time, the TransactionManager will store your context in the persistent space and will re-queue it at regular intervals (default to 5 seconds, configured by a retry-interval property) back into the main queue. Imagine you have a system that connects to different remote processors and one processor goes slow or down for a while. Even if you are handling a big number of concurrent sessions, eventually all sessions might get locked waiting for responses coming from that particular endpoint. You could solve a situation like this by using multiple transaction manager instances (one per endpoint) but with this new RETRY thing, you can just RETRY the transaction at a later time (usually a few seconds later). You can place a retry counter in the context, or check the timestamp in order to avoid retrying the transaction forever. There´s an example in jpos6/modules/txnmgr/test/org/jpos/transaction/

Amazing visit

/by apr/ Yesterday´s late afternoon, I had an unexpected and welcome IM call from my friend Rodolfo Pilas, he said he was there with Mr. Federico Heinz and they were keen to step by Would you believe that? Interesting enough, I was aware Mr. Heinz was coming to speak at the JIAP 2006. That´s an event organized by government employed IT people. I would have loved to assist to Mr. Heinz presentation, but something very funny and weird happened to me last year with the JIAP 2005 event, so I said nah… I hate to speak at events (just because I get very anxious due to lack of practice) but last year, JIAP organizers asked the local Linux User Group about providing a presentation about local free software project initiatives, and they contacted me to speak about ours. Instead of talking about jPOS, I thought that it would be a good idea to speak about how we develop software in a distributed and collaborative environment, trying to map this to distributed government agencies with their replicated IT departments, reinventing the wheel on a regular basis instead of collaborating with each other. I said yes, but then the organizers though that talking at their event would somehow promote jPOS, so they thought it was a good idea to charge me! Hilarious. I asked for a printed and signed quote just to frame it and nail it at my office, but never managed to get one. I was told that the audience was mostly mainframe guys, I think that they really need to listen about how we develop software these days or they´d be lost, but yet the people that´s leading them are greedy and blind and driven just by money. Anyway, I did not assist to Mr. Heinz´s presentation yesterday, but was honored by his visit to our little jPOS project, it was amazing to be able to speak with a person of his caliber, and to get to know that he likes our logo! Thank you for your visit!


/by apr/ jPOS Spaces have proven to be very useful for us and I believe that they could be useful for others too, but adding a full jPOS dependency may prevent its use by other projects, so I did some minor surgery in order to remove a few dependencies and I’ve created a minimalist version of it here. If you think I’m doing this just to test the new exciting Google Code hosting you are wrong, but I have to admit that it was the trigger… I couldn’t resist to give it a shot :) A note to jPOS users: You don’t need miniSpace, you have it inside jPOS. I believe that we can benefit from external feedback from people using miniSpace - we will port improvements made in miniSpace-land back to jPOS-land as we see fit.


/by apr/ Complex machinery We recently went live on yet another instance of a jPOS based application called OLS.Switch, and after several months working with the OLS team we reached over 1000 iterations. At the same time, we see a lot of people in our mailing lists, new to jPOS, sometimes new to payment systems, and still facing a deadline in say … one week (two weeks for the pragmatists). I consider myself an average programmer, but I’ve been exposed to payment systems for over 15 years, and I wrote jPOS from scratch, and the OLS team has tremendous experience in this area, and we still need to work month after month, iteration after iteration in order to get ready to go live. Am I missing something? Is this just because I don’t use an IDE? Hmm… I doubt it. Andy (OLS’ CTO) put it in a very nice way in his blog.


by apr on certifications


A recent post in the users’ mailing list has triggered this post.

The post goes something like this: “The problem was they where modifing this server (it’s a test server) and that’s why I didn’t get anything.” Sounds familiar?

This happened to me dozens of times.

These people think they are smarter than us developers, so they use our time to test their systems, and they hide behind their big names. I can imagine they say something like “hey, let’s test this new patch that handle our latest features with the next stupid programmer that needs to certify”.

They do test driven development, but the problem is YOU (developer) are the tester, and by the way, you don’t know that.

They hide behind their big brand names and you get to listen things like “Your system has to be failing, our’s process a massive amount of transactions every single day, we just can’t afford to go wrong”.

And that’s very true, their production system, installed turn-key by their solutions provider actually works (until they start messing with it), but the little crap that they have put together for your tests, running some bogotified version of the original software with some spaghetti-patch to support the latest feature recently requested by the marketing division never worked before.

It would be okay if they are honest and tell you “Listen, test with this proven certification environment, and then, if you don’t mind, we would like to run a couple of tests in our new system”.

That’s fair, you usually need a little bit of help during certification and it’s very nice to provide some useful feedback, but the problem is they usually don’t get to acknowledge their system has never run before until you spend a week, testing day after day, frustrasted against a wall.

UPDATE: the previous title was rude so I changed it to BCFH (based on BOFH, C for certifier)


/by apr/ This time we got it right! I’ve tagged a v1_5_2 version and moved the version number to 1.5.3, but while working on these major build system changes I thought it was time go up to 1.6.0. We’ll refer to jPOS 1.6.x versions as jPOS 6 and its svn repository is I was thinking about doing these changes on the current ‘jpos’ directory, but looking further into the work we have ahead, it seems like it will require several days or weeks to have everything migrated to the new environment, so a ‘jpos6’ temporary directory seems like the way to go. Once we are ready to release ‘jpos6’, we can get rid of the older versions in ‘trunk’ (it will remain available in tag/v1_5_2/) I’m committing a basic installation there with just a the main ‘jpos’ and ‘q2’. The nice thing about the new layout is that we don’t need conditional compilation anymore, if something requires some extra library, it will be distributed in its own module. I will update the README and INSTALL files as time permits (help is welcome), but to get an idea of where we’re going, you may want to read the first chapters of the jPOS-EE SDK guide. If you want to stay up to date with the latest commits, you can subscribe to the SVN notifications list (send e-mail to jpos-commits-subscribe at jpos dot org). *This time we got it right!** (you’ve got to read that piece, it’s so funny and sad at the same time that everything there is sooooo true…)

Specs Specs Specs

/by apr/ I work with Andy in several projects and I have to tell you, he’s the most detail-oriented person I’ve ever met. He’s the kind of guy that enjoy details, gets excited reading and writting specs and I don’t know how, but he seems to memorize all of them. He is a voracious spec reader® that can pinpoint potential issues in specs from major institutions –that went through multiple internal revisions– at first sight, and he writes amazingly detailed specs for even the simplest task. During development, when we ask for some piece of information, his invariable answer (even from Blackberry) is: ‘look at page xxx of my spec’ (we have a roadmap spec that includes pointers to all other involved specs, he calls that the ‘master project workbook’ where we can find literally everything related to a given project). Andy has a blog now, and interesting enough, his blog has an specs section that I’m sure readers of this blog will find very useful. I highly recommend that you add his feed to your agregator.

Encrypting sensitive data

/by apr/ Protecting Customer's Data

Many jPOS-EE users and PEP members have a recurring problem: encrypting sensitive data in order to protect their customers and if possible, make auditors happy.

But there’s a recurring issue: unless you use an HSM, a smartcard to protect your keystore, or you have someone enter a pass-phrase at boot time, you have to live with the fear that someone could somehow access your keystore, get your master keys and decrypt your sensitive data.

The problem becomes more critical in situations where jPOS is installed at small merchants or medium-sized store chains, we can’t ask them to enter a passphrase at boot time, we don’t have smartcard readers there and we don’t have HSMs.

We could do some security through obscurity but we leave that to our competitors :) Based on the work performed by Hani in the jPOS’s security module, and the DUKPT implementation we did with Dave from InfoSecBlurb in a related project, I came up with a nice solution that I’d like to present here in order to get your feedback. Here is a simple example:

SSM ssm = (SSM) NameRegistrar.get ("ssm");
SecureKeyStore ks = new SimpleKeyFile ("cfg/keys.cfg");
SecureDESKey bdk = (SecureDESKey) ks.getKey ("our-bdk");
String encrypted = ssm.customEncrypt (bdk, "4111111111111111=0805".getBytes());
String decrypted = new String (ssm.customDecrypt (bdk, encrypted), "ISO8859_1");

customEncrypt produces something like this:


and customDecrypt gives you your original byte[] array back. Here is what we do:

  • We generate three clear-text 128-bit keys
  • We use the security module’s console to enter those keys and produce a BDK encrypted under the local master keys (using standard key-ceremony procedures)
  • The BDK cryptogram is stored in the keys.cfg file
  • When we receive a message to encrypt, we generate two 64-bit random DES keys that we use as KSNs. We split them to produce a baseKeyID, deviceID and transactionCounter as required by KeySerialNumber
  • We use our BDK and these two KSNs to produce a pair of derived 64 bit keys
  • We combine these two 64 bit keys and do 3DES with them in order to encrypt our payload
  • We use both KSNs and derived keys plus the encrypted payload to produce an SHA hash, so we can perform sanity checks

So far, so good, it seems pretty nice, but it’s still breakable if you have access to the master keys and the BDK’s cryptograms, or, if you have access to the jar and you manage to call ssm.customDecrypt with the right parameters.

In order to mitigate that, I’m experimenting with several anti-debug features; besides obfuscating the jpos-ee jar (that’s something that the PEP version already has), we are trying to have a hash of the stacktrace of the caller in order to validate the caller id and we have some timing control too.

But anyway, here is the best thing, a very simple one: we don’t have to run this at every merchant, we can use this approach (or an HSM-based one) at a central location (such as an acquirer or transaction switch). When an acquirer or transaction switch receives say a 0200 message with a some sensitive data (i.e. PAN, EXP, TRACK1, TRACK2), it could reply with a 0210 carrying a cryptogram of that data computed at the remote facility, where we don’t have access to the LMKs. We could do that with an extremely simple ISOFilter. Imagine something like this: Message sent from merchant to acquirer:

  <field id="0" value="0200"/>
  <field id="3" value="000000"/>
  <field id="4" value="000000012183"/>
  <field id="7" value="0309123613"/>
  <field id="11" value="010370"/>
  <field id="12" value="123613"/>
  <field id="13" value="0309"/>
  <field id="22" value="022"/>
  <field id="24" value="105"/>
  <field id="25" value="00"/>
  <field id="35" value="4111111111111111=0805"/>
  <field id="41" value="29110001"/>
  <field id="42" value="012345         "/>
  <field id="44" value="0000000000"/>
  <field id="48" value="0010000000000000"/>
  <field id="49" value="858"/>
  <field id="60" value="jPOS-1.5.1"/>
  <field id="62" value="7297873"/>

Message response from acquirer:

  <field id="0" value="0210"/>
  <field id="3" value="000000"/>
  <field id="4" value="000000012183"/>
  <field id="7" value="0309123613"/>
  <field id="11" value="010370"/>
  <field id="12" value="123613"/>
  <field id="13" value="0309"/>
  <field id="22" value="022"/>
  <field id="24" value="105"/>
  <field id="37" value="000000130682"/>
  <field id="38" value="130682"/>
  <field id="39" value="00"/>
  <field id="41" value="29110001"/>
  <field id="42" value="012345         "/>
  <field id="44" value="0000000   0000000"/>
  <field id="48" value="0010000000000000"/>
  <field id="49" value="858"/>
  <field id="62" value="7297873"/>
  <field id="63" value="B9B35297F48C792646CED3B367F40E6D,F302BA266FDAC3A6D4773B4C74553C49F584934BCA9224F8,5C9538880BED87FD"/> 

The acquirer’s outgoing filter would build a custom cryptogram for us either using the aforementioned algorithm, an HSM on any other similar approach, and it would put it in a private field for us (in this case we used field 63, but could be any other free variable length private field) and an incoming filter at the acquirer would take care of decrypting it locally, unsetting that private field and setting the appropriate field 35 (same goes for other sensitive fields such as 2,14,45,52,55, etc.).

At the client application, you could store some clear-text information (such as BIN, last four digits of the PAN) for reporting purposes and store that opaque cryptogram produced by your acquirer/processor; that would be your ticket to request funds for that transaction if for some reason you need to perform a batch upload operation.

Although an scheme like this wouldn’t solve problems like this, it would certainly help merchants protect its customers’ data with a minor effort from the acquirers. The encrypted information could be an inner ISOMsg with just some sensitve field.

The document Secure Payment Framework presents some related concepts. I’d love to listen to potential flaws in this. I’d also like you to exchange some anti-debug ideas and links. So far, I’m serializing a Throwable’s stacktrace, producing and validating a set of hashes with valid caller id info, an approach that is extremely sensible to JVM versions…

Small World

/by apr/ For some time, my brother in law, who lives in Brasil, has been talking to me about his girlfriend’s sister’s husband (aka his brother in law) and the fact that he was a brilliant programmer, a genious! I went ‘oh yeah… great’ but honestly, I didn’t pay too much attention.

Recently, he came to our place to spend with us Dec 31th, and he said that his girlfriend, along with her sister and husband (the brilliant programmer) were keen to come too, but they couldn’t find a hotel (we were at a resort town, fully booked for the holiday season).

While I was cooking a BBQ , he went talking about how the guy has moved to the US, that he gives keynotes in many conferences, etc. etc. and after a few hours, he said, you might know him, his name is Miguel de Icaza.


I was shocked! I could have booked a hotel 6 months in advance, we could have hosted them at our place!

Damn, I missed this great oportunity to shake hands with a programmer of the caliber of Miguel! Lesson learnt, do not > /dev/null all what your *-in-law family members have to say … :)

On the bright side, Miguel didn’t have the chance to visit Punta del Este (a very nice city), but he saved himself a long rap about jPOS … I have a Mono T-shirt now, sent by the author, and I expect him to get his jPOS T-shirt soon (on its way…) Small world, huh?