/by apr/

I´ve spent last week working with Andy and an amazing team of top-notch developers that are building what I think is the biggest jPOS application ever. Code-named "The Gladiators Team", this group is developing a next-generation acquiring switch capable of delivering 900 tps continuous throughput. I was glad to see they are pushing jPOS to its limits using the latest and greatest features such as the TransactionManager´s continuations support. We found and fixed some nasty bugs that were not showing up under light load, there´s now a Context.resume() operation (you no longer need to re-queue a paused transaction), paused transactions can now have an expiration (they get automatically continued by the transaction manager) and we worked out a new interface for the MUX that will be able to operate asynchronously by using the new ISOResponseListener interface. They have some really good requirements related to system monitoring, realtime re-configuration and stats collecting so you can expect changes in jPOS in that front during the following weeks. (you can follow the changes in the jpos-commits feed). Working with this team for a full week was a learning experience, I learnt from their system architect (a very realistic, YAGNI-oriented type) that they use stochastics in their load simulation, a technique that allows them to simulate a production environment in a very realistic way. This makes a lot of sense, you don´t usually get 900 tps from a single source, you get connections from thousands of different merchants and dozens of acquirers simultaneously producing different kind of loads. You can expect some improvements in our clientsimulator in the future. This is not a 900 tps peak system, it is a 900 tps continuous system (they can probably peak at way above the 2000 tps mark). I got word yesterday that they managed to run the system at full 900 tps for 12 hours (not bad, huh?). The good news: four members of this team are now jPOS committers and their company is extremely open-source friendly, so we can expect great contributions from them in the near future. Ansar, Tushar, Manoj and Vishnu, Welcome!. I´m looking forward to see you again guys, (specially on Fridays :))

/by apr/ For those of you using jPOS-EE' Web User Interface you''ll be happy to know that Florin is doing CPR to JPublish, the brilliant framework that we use in jPOS-EE. We are excited to know JPublish will continue to be supported and I can't wait to get my hands dirty with the new JPublish+DWR integration. For those of you looking for a clean and elegant web framework, look no further, go visit JPublish's new home. Kudos Florin!

/by apr/

Hugo van der Zee (with the Cyclos project) pointed me to this nice Google TechTalk Video from the subversion guys.

We are a tiny little project compared with giant projects such as subversion, and we are lucky that we don´t have the problems that they describe, but it´s good to be aware of them before they happen. Interesting enough, I´ve identified the power-plant problem a couple of times through these years ("Hey, my new version of jPOS is almost ready!! I will send it for your review soon -- I´ll call it jPOS 7).

Understanding that this is actually a problem is a good first step to solving it.

/by apr/ jPOS 1.6.0 (aka jPOS 6) represents almost two years of hard work that include bugfixes, performance tuning and new components as described in our ChangeLog as well as a completely redesigned component-oriented build environment pioneered by the jPOS-EE project. jPOS 6, distributed under a GPLv2 license, is considered a stable release and is being used by Fortune 500 companies to process millions of transactions per day. You can download it here. We will move our development version to 1.6.1.

/by apr/

I'm happy to announce a new and exciting TransactionManager's feature -- drum roll... PAUSE... :) ... CONTINUATIONS!.

You can now PAUSE a transaction (at a participant boundary) by returning PREPARED|PAUSE (or ABORTED|PAUSE).

The transaction gets suspended and resumes once you place it again in the TransactionManager's queue (i.e. by calling txnmgr.queue (Context)).

Why this is important to us?

Imagine you have a transaction manager configured to run 100 simultaneous sessions.

Even if 100 is a big number, when you depend on external components (such as a remote host) that can go slow you can easily exhaust all sessions.

When you have a single remote host and it goes slow, there's nothing you can do about it, but when you connect to multiple acquirers and one of them goes slow, you don't want to penalize the rest.

This problem has been described here where we use multiple TransactionManagers (one per remote host) with a pre-configured reasonable number of sessions and we forward transactions from one transaction manager to the next.

With continuations, we can split a participant that has to wait on remote events (i.e. a QueryHost or QueryHSM kind of participant).

The first one just sends the request to the remote system and returns PREPARED|PAUSE.

The session gets free and ready to process new transactions.

Once we receive a response, we just queue the Context back into the TransactionManager (I'll add a Context.resume() method to ease finding the right space and queue) and the transaction continue processing.

/by apr/ A nice side effect of going up the food chain (in addition to getting tastier food) is that you get to interact with really smart and knowledgeable people looking for elegant solutions to complex and challenging problems.

We have production systems now doing 600k txn/day, but we are now shooting for 2M/day.

We have systems that can do 300 sustained TPS for 30 minutes, but we face a requirement to do 2000 TPS sustained for five hours now.

In a recent meeting while Andy was doing a walk-through of their OLS.Switch TransactionManager's configuration, we where asked if the participants could run in parallel.

Although we tried to explain that the role of the TransactionManager was to assemble the flow of the transaction in an "execution chain" where the order had to be preserved, one the engineers there actually had a very good point. There are many situations where you want to run things in parallel, for example, you can do some database validations while firing a pin translation message to the HSM pool.

On a multi-database environment, you may want to simultaneously fire different queries to different databases. I liked the idea and started to work as soon as the meeting was over. With the new org.jpos.transaction.participant.Join participant, you can convert something like this:

<participant class="com.your.company.DoPinEncryption">
   ...
</participant>
<participant class="com.your.company.QueryDatabase">
  ...
</participant>

into:

<participant class="org.jpos.transaction.participant.Join">
 <participant class="com.your.company.DoPinEncryption">
   ...
 </participant>
 <participant class="com.your.company.QueryDatabase">
   ...
 </participant>
</participant>

just by wrapping them inside the Join participant and run both in parallel (you are not limited to just two inner participants, you can place there as many as you want). Join honors the AbortParticipant contract so you can have your inner participants called even if the transaction is meant to abort.

I had to do some minor changes to the TransactionManager implementation as we needed a reference to the QFactory in order to reate the inner participants. In order not to break existing participants and to avoid adding a new interface, the TM uses Reflection in order to attempt to push a reference to the TransactionManager to the participant at init time. If your participant has a method:

   public void setTransactionManager (TransactionManager mgr);

then it would get called. You can use that reference to pull a reference to the QFactory which in turns uses the the MBeanServer to instantiate the participants using an appropriate dynamic classloader. If you are still not using the TransactionManager in your jPOS application, you should consider giving it a try, we believe it's the way to go.

/by apr/ I have been introduced to Strohalm NGO and the Cyclos project a long time ago by Todd Boyle but today I have been honored by a courtesy visit of Cyclo's project lead, Mr. Hugo van der Zee. I've seen an impressive demo of the upcoming Cyclos3 and Hugo was introduced to jPOS and miniGL. We found many areas of collaboration:

• Cyclos is going to support [m]POS transactions. jPOS is a natural fit there.
• Cyclos has its own internal ad-hoc accounting system. They will seriously evaluate the posibility of moving to the more generic miniGL.
• Both projects are highly aligned regarding the tools we use, the license we use and complement each other pretty well.

Hugo and I will be working in the integration but we are also looking for an skilled jPOS developer in our area to work full time in it (interested parties please contact us). Besides the technical aspects of Cyclos and jPOS, their implementations of community currencies are amazing and something really good for weak economies. Hugo has a background in economics and I was lucky to get a first-hand explanation of the rationale behind them. I was proud to know that our government through our major bank BROU sponsored by EMPRETEC are actively supporting an implementation here. I´m sure the collaboration and integration will be good for both projects, so Welcome Cyclos!.

/by apr/ As a follow-up to the recent post about miniGL layers we have removed support for foreign amounts at the transaction.entry level, foreign currencies are now handled in their own layer. That means that if you have a checking account and a savings account (we usually figure out the account type based on the content of the processing code field) in multiple currencies, you don´t need a separate set of accounts (one per type), the same account can hold transactions in multiple currencies. As mentioned in the previous post, we use layers to deal with pending versus real transactions (think pre-auth/completion) and we found an easy way to add multi-currency support by defining a PENDING_OFFSET constant. We are still working in the layers plan for our jCard system, but as an initial take, we have defined the PENDING_OFFSET=1000 and we store transactions in their corresponding layer using the iso4217 currency code, so e.g.: we store real dollars in 840 and pending dollars in 1840 (same goes for other currencies). So far the new setup looks extremely simple and straightforward. A close friend and CPA has contributed a very nice idea to the mix, conversion layers. I´m still not sure if we want to handle them with the existing database schema or we want some supporting tables for that, it has been good food for thought for now.

here is a very nice article by Andy about implementing an ISOPackager. Highly recommended if you are new to jPOS and you are trying to figure out how to connect to a remote host.

/by apr/ I´ve added a new feature to miniGL: Layers.

This new feature is designed to support multiple balances for a single account, such as available versus accounting balance.

We could use some flags for unsettled transactions, or pre-auths waiting for completion, but with this new layers support everything gets more simple, reusable and IMO elegant. Imagine a simple ATM withdrawal transaction:

(both liabilities as seen from an issuer perspective, fees not included here for simplicity)

Sounds very nice, but we have a problem, this transaction is still unsettled until we get a batch close or completion transaction, so if we don´t have this new layers feature, we would end up having either some kind of flag or duplicating accounts, i.e:

• Cardholder´s checking account
• Cardholder´s checking account (pending)
• ATM Network account
• ATM Network account (pending)

With our new layered approach, we post real transactions to layer zero (ground) and pending transactions to say layer one. When the transaction gets settled at the end of the day, we just reverse the effect of the transaction in layer one (blue) and post it to layer zero (black), e.g, when we get a pre-auth, we could post:

Pre Authorization

(In this example, '1' would be the 'pending' layer).

Completion

(Now it´s confirmed, we "ground" it)

I´ve modified the existing rule implementations in order to support this new layers feature, e.g. the DoubleEntry rule (that verifies that debits equals credits in the reporting currency for a given GLTransaction) now accepts a list of layers to check, so we can be very strict in some layers (such as the ground layer aka the real deal) while more relaxed in other ones. minigl.dtd has been modified in order to support a new <layers> element in rule definitions.

Some other uses for this include data-entry application where a junior clerk could post transactions to a given layer for later verification, then someone else with enough karma to authorize the transaction would "ground" it. We can use this feature to track "budgets", "cost centers" or just to ease some rule implementations (e.g.: we can have minimum balances layer, a max balances layer, and so on).

The layer number is arbitrary to a given miniGL application. On multi-currency systems, we use the currency ISO number as the accounting layer for a given currency.

When you ask for a balance for a given account at a given journal, miniGL gives you the ability to get the balance for a set of layers. This comes very handy when computing accounting versus available balances in ATM and POS transactions. You ask getBalance(journal, account, 0) and you get the accounting balance. You ask for getBalance(journal, account, 1) and you get the pending transactions (still unsettled). But if you ask getBalance(journal, account, 0,1) miniGL combines both layers to give you the available balance. We also use a third layer for overdrafts (used in credit card applications), say layer '2', so on certain transactions that would allow an overdraft (i.e. POS purchases), we could check getBalance(journal, account, 0,1,2).

I just hope the guy in the mail stamp up there is not twisting in his crave ... :)