ISO 8583:2023 Datasets in jPOS

Since ISO 8583:2003, the standard has defined three types of data elements: primitive, constructed, and composite. Primitive and constructed fields have always had first-class support in jPOS. The third type—composite fields—has historically been treated as opaque binary blobs, with application code responsible for parsing their contents manually.

jPOS 3.0.2 changes that.

Background: what composite fields are

ISO 8583:2023 defines several data elements—DE-034, DE-043, DE-049, DE-055, DE-071, DE-104, and others—as composite fields. Rather than having a fixed structure, a composite field contains one or more independent sections called datasets. This design lets a single field carry different categories of information in a flexible, extensible way.

Each dataset follows one of two encoding formats:

TLV datasets (identifier 0x01–0x70): sub-elements are encoded as BER-TLV tag-length-value pairs and can appear in any order. Multi-byte tags following ISO/IEC 7816-6 conventions are supported.

DBM datasets (identifier 0x71–0xFE): sub-elements are indexed by a second-level bitmap (similar in structure to the message bitmap). A DBM dataset can also carry a TLV continuation for rarely-used elements not covered by the bitmap.

Each dataset is preceded by its one-byte identifier and a two-byte big-endian length, giving the full wire structure:

[ id (1 byte) ][ length (2 bytes, big-endian) ][ content ]

DE-055 (ICC/EMV data) is a special case: it carries raw BER-TLV directly, with no dataset identifier or length envelope.

What jPOS now provides

Core classes:

• Dataset — interface representing a single dataset
• ISODataset — mutable implementation with fluent builder support
• DatasetElement — holds one decoded sub-element
• ISODatasetField — top-level field component that holds one or more datasets

Packagers:

• DatasetPackager — reads and writes composite fields with full TLV and DBM support
• ICCDataPackager — special-case packager for DE-055 raw BER-TLV

Message API additions:

• ISOMsg.with(field, value) — set and return this for fluent chaining
• ISOMsg.without(field...) — unset and return this for fluent chaining
• Path-based access for dataset elements via msg.with("55.0x9F26", bytes) / msg.without("55.0x9F26")

Packager:

• cmfv3.xml — a CMF packager that enables dataset-aware handling for DE-034, DE-043, DE-049, DE-055, DE-071, and DE-104. The original cmf.xml is unchanged for backward compatibility.

Building a message

With cmfv3.xml, you can build and parse composite fields using the same dot-path style used for nested sub-messages:

GenericPackager packager = new GenericPackager("jar:packager/cmfv3.xml");

ISOMsg msg = new ISOMsg("0100");
msg.setPackager(packager);

// Dataset paths are: field.datasetId.elementId
msg.with(3,   "000000")
   .with(11,  "123456")
   .with(41,  "TERMID01")
   // ICC data (DE-055): field.elementTag — no dataset wrapper for EMV
   .with("55.0x9F26", ISOUtil.hex2byte("1122334455667788"))
   .with("55.0x9F10", ISOUtil.hex2byte("06011203A0B800"))
   .with("55.0x9F36", ISOUtil.hex2byte("0022"))
   .with("55.0x95",   ISOUtil.hex2byte("0000000000"))
   // Verification data (DE-049): TLV dataset 0x01
   .with("49.0x01.0x5F2A", ISOUtil.hex2byte("0840"))
   // Verification data (DE-049): DBM dataset 0x71
   .with("49.0x71.1", "1")
   .with("49.0x71.2", "1234");

byte[] packed = msg.pack();

Reading values back after unpacking:

ISOMsg unpacked = new ISOMsg();
unpacked.setPackager(packager);
unpacked.unpack(packed);

ISODatasetField field55 = (ISODatasetField) unpacked.getComponent(55);
ISODataset icc = (ISODataset) field55.getDataset(55); // keyed by field number for ICC

byte[] cryptogram = icc.getBytes(0x9F26);  // Application Cryptogram
byte[] iad        = icc.getBytes(0x9F10);  // Issuer Application Data

ICC data backward compatibility

If you already have ICC data as raw TLV bytes from the legacy cmf.xml packager, cmfv3.xml produces byte-for-byte identical output. This is verified directly in the test suite:

// Legacy CMF: set DE-055 as raw bytes
ISOMsg legacyMsg = new ISOMsg("0100");
legacyMsg.setPackager(new GenericPackager("jar:packager/cmf.xml"));
legacyMsg.set(new ISOBinaryField(55, rawTLVBytes));
byte[] legacyPacked = legacyMsg.pack();

// CMFv3: set DE-055 using the fluent path API
ISOMsg datasetMsg = new ISOMsg("0100");
datasetMsg.setPackager(new GenericPackager("jar:packager/cmfv3.xml"));
datasetMsg.with("55.0x9F26", ISOUtil.hex2byte("1122334455667788"))
          .with("55.0x9F10", ISOUtil.hex2byte("06011203A0B800"))
          .with("55.0x9F36", ISOUtil.hex2byte("0022"))
          .with("55.0x95",   ISOUtil.hex2byte("0000000000"));
byte[] datasetPacked = datasetMsg.pack();

assertArrayEquals(legacyPacked, datasetPacked); // identical wire bytes

Existing systems continue to interoperate without any changes.

Clone safety

ISOMsg.clone() now deep-clones dataset fields, so modifying a cloned message does not affect the original. The without() method also auto-removes the parent field when all its dataset elements are cleared:

ISOMsg clone = (ISOMsg) original.clone();

clone.without("55.0x9F10")           // remove one ICC element
     .with("55.0x95", zeroes);        // add another

// Field 55 in clone is independent of field 55 in original
assertNotSame(original.getComponent(55), clone.getComponent(55));

Getting started

Switch your packager from cmf.xml to cmfv3.xml:

<bean id="packager" class="org.jpos.iso.packager.GenericPackager">
  <property name="configuration">
    <value>jar:packager/cmfv3.xml</value>
  </property>
</bean>

Existing code that treats composite fields as opaque binary fields continues to work unchanged with cmf.xml. No migration is required unless you want to take advantage of structured access.

What's next

The DatasetPackager is designed to be subclassed. You can define custom packagers with their own set of bitmap sub-elements for any composite field used in a specific network or scheme.

DE-018 (Message Error Indicator), which uses the same dataset addressing model to report parsing errors, will be wired up in a subsequent release.