Two incomplete fixes in IBM’s Z cryptographic stack

When a memory-safety fix lands, it lands on the path that triggered it. The same mistake, written the same way a few functions over, usually stays. Two recent fixes in the cryptographic software for Linux on IBM Z were read that way, and in each case a sibling of the bug was still live: eight out-of-bounds reads in opencryptoki’s ASN.1 key decoding, and a stack-buffer overflow in openssl-ibmca’s EC key handling. Both were reported with a reproducer and a patch, and both were fixed and credited by the maintainer.

The method

A published security fix is a map of a mistake the project has just agreed it made. The fix itself shows the exact idiom to look for; the diff shows exactly how far the author looked. The gap between the two — the places the same idiom appears that the diff did not touch — is where the next finding usually is. Neither of these was a new class of bug. Each was the existing, just-patched bug, still present one path over.

opencryptoki — eight out-of-bounds reads in the key-type checks

opencryptoki is the PKCS #11 implementation for Linux on IBM Z. Applications call it to reach the platform’s cryptographic hardware — the CCA and EP11 coprocessors, the CPACF instructions, a software token — through a standard interface. It decodes keys that cross that interface in BER/DER, the ASN.1 encoding, and some of those keys arrive from outside.

The parent bug, CVE-2026-40253, was an out-of-bounds read in that decoder. Its fix (commit ed378f4, in 3.27.0) threaded a buffer length into the five primitive decoders so each could check it had the bytes it was about to read. Correct, as far as it went. But the decoder also has a set of identify checks that run first — small memcmps that compare a fixed OID or algorithm-identifier prefix against the front of a decoded, attacker-sized AlgorithmIdentifier to decide which key type this is. Those were not given the same length precondition:

/* usr/lib/common/asn1.c — one of eight identify checks */
if (memcmp(alg, ber_idEC, ber_idECLen) != 0)   /* reads ber_idECLen bytes */
    return CKR_FUNCTION_FAILED;                 /* with no check that alg is that long */

If the decoded algorithm field is shorter than the prefix being compared — an empty AlgorithmIdentifier (30 00), say, or an empty OID (06 00) — the memcmp reads off the end of the allocation. Eight sites across the classic key types had this shape; four were reproduced under AddressSanitizer against the real decoder. Two of the eight used a second idiom — a memcmp against a decoded template rather than a literal — that a grep for the OID constants missed, and that the fuzzer and a CodeQL query caught.

The path is reachable where opencryptoki decodes a key it did not create: C_UnwrapKey for a wrapped private key, SubjectPublicKeyInfo import for a public one. The attacker pads the version integer so the short algorithm identifier sits at the end of the allocation, which is what turns the over-read into a fault instead of a quiet read of the next heap object.

Impact. Denial of service of the token via an out-of-bounds read — the same ceiling as the parent CVE, and the same Moderate band. It is a read of a fixed number of bytes compared against a constant, so it is a one-bit oracle at most, not a usable memory disclosure. Credited, not a paid finding: IBM’s PSIRT holds CVE assignment and none was issued for this follow-up.

The fix adds the missing length precondition at all eight sites, and it was instructive in one detail. My first patch for the RSA public-key check compared the full template length, which would have rejected valid RSA public keys — the SubjectPublicKeyInfo path returns the OID length there, not the full algorithm-identifier length including its NULL parameters. Ingo corrected that one site to compare only the OID portion, took the other seven as written, and added two further hardening commits on top. A crash-only reproducer would never have caught the over-strict guard; a fix needs a valid-input-still-succeeds check as well as a malformed-input-is-rejected one. That lesson is now part of the pre-send routine.

openssl-ibmca — a stack overflow in EC key handling

openssl-ibmca is the OpenSSL engine and provider that routes OpenSSL operations to the same IBM Z hardware through libica. Its parent fix (commit 4bf3166, reported by another researcher, no CVE) closed a stack-buffer overflow on the signature-verify path: a crafted DER signature with oversized r/s values made a length calculation underflow and overrun a fixed stack array. The diff touched verification only.

The identical calculation appears for the EC private key, in ibmca_ec_make_ica_key(), and was not guarded:

/* src/ibmca_ec.c:158 — D is a fixed IBMCA_EC_MAX_D_LEN stack buffer */
n = privlen - BN_num_bytes(bn_d);   /* privlen unsigned; underflows if d is longer */
memset(D, 0, n);                    /* n becomes a huge size_t */
BN_bn2bin(bn_d, &D[n]);

Give it an EC private key whose scalar d is longer than the curve’s field size, and privlen - BN_num_bytes(bn_d) goes negative, wraps to an enormous size_t, and the memset runs off the stack array. This path is reached from ECDSA signing and from ECDH key derivation, i.e. whenever the engine uses the key.

Impact. A crash, denial of service. This is calibrated deliberately: the oversized memset faults before the controlled BN_bn2bin write is ever reached, so there is no attacker-controlled write here — it is an abort, not a route to code execution. The attacker model is also weaker than the verify-path parent: it needs the engine to sign or derive with an untrusted private key, which is real for a key-import or signing-gateway configuration but is not the default posture.

It was reproduced without IBM Z hardware. A standalone AddressSanitizer harness replays lines 155–160 verbatim against a real D[66] buffer and loads a crafted key through d2i_ECPrivateKey; OpenSSL’s core accepts the oversized scalar (it only rejects on serialisation, which the sign path never does), so the engine receives the raw d. A normal 32-byte scalar gives n = 0 and is clean; a 40-byte scalar gives n = -8 and AddressSanitizer reports a negative-size memset on the stack object. The fix in PR #132 adds the bound check and sweeps the x and y siblings in both ibmca_ec_make_ica_key() and ibmca_ecdh_compute_key() — the whole class, not only the reported site. Ingo noted that the engine interface is deprecated in favour of the provider, and fixed it anyway.

Why they were still there

A length check added at the front of a parser is correct for the code as it stands, and goes stale as handlers are added behind it that read further than the check covers. A fix written for the verify path leaves the import path untouched because the author was looking at a verify crash. In both cases the project did the right thing for the bug in front of it; the method here is just to assume the right thing was done once, in one place, and to go and look at the others. The tell is always the same idiom — a fixed-size read or compare against a length that attacker input can make too small.

Disclosure

Fix

For opencryptoki, upgrade to 3.28.0 or apply PR #979 to 3.27.0. For openssl-ibmca, apply PR #132, or — the maintainer’s own recommendation — move from the deprecated engine to the provider interface. All of the changes are added length checks and reject only malformed input; valid keys and signatures are unaffected.