ios7 - RSA: encrypt in iOS, decrypt in Java

Question

I have a public key that's sent from a Java server. The base64 encoded strings match before I decode and strip out the ASN.1 headers. I store the public key in the keychain with SecItemAdd.

So I'm trying to encrypt the data using the public key and decrypt it with the private key in Java. I'm using SecKeyEncrypt on the iOS side and Cipher on the Java side.

What I'm encrypting is the symmetric AES key that encrypts my actual data, so the key length is 16 bytes. When simply base64 encoding the key, everything works, so I know something is wrong with this RSA encryption.

Here's an example of my iOS call:

OSStatus sanityCheck = SecKeyEncrypt(publicKey,
        kSecPaddingPKCS1,
        (const uint8_t *) [incomingData bytes],
        keyBufferSize,
        cipherBuffer,
        &cipherBufferSize
);

Here's an example of my Java call:

public static byte[] decryptMessage (byte[] message, PrivateKey privateKey, String algorithm) {
    if (message == null || privateKey == null) {
        return null;
    }
    Cipher cipher = createCipher(Cipher.DECRYPT_MODE, privateKey, algorithm, false);
    if (cipher == null) {
        return null;
    }

    try {
        return cipher.doFinal(message);
    }
    catch (IllegalBlockSizeException e) {
        e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
        return null;
    }
    catch (BadPaddingException e) {
        e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
        return null;
    }
}

private static Cipher createCipher (int mode, Key encryptionKey, String algorithm, boolean useBouncyCastle) {
    Cipher cipher;

    try {
        if (useBouncyCastle) {
            Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider());
            cipher = Cipher.getInstance(algorithm, "BC");
        }
        else {
            cipher = Cipher.getInstance(algorithm);
        }
    }
    catch (NoSuchAlgorithmException e) {
        e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
        return null;
    }
    catch (NoSuchPaddingException e) {
        e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
        return null;
    }
    catch (NoSuchProviderException e) {
        e.printStackTrace();
        return null;
    }

    try {
        cipher.init(mode, encryptionKey);
    }
    catch (InvalidKeyException e) {
        e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
        return null;
    }

    return cipher;
}

I've tried so many combinations and nothing has worked.

• iOS: PKCS1, Java: RSA/ECB/PKCS1Padding
• iOS: PKCS1, Java: RSA
• iOS: PKCS1, Java: RSA/None/PKCS1Padding (throws org.bouncycastle.crypto.DataLengthException: input too large for RSA cipher.)
• iOS: OAEP, Java: RSA/ECB/OAEPWithSHA-1AndMGF1Padding
• iOS: OAEP, Java: RSA/ECB/OAEPWithSHA-256AndMGF1Padding

I've also tried using the internal Java provider as well as the BouncyCastle provider. The javax.crypto.BadPaddingException gets thrown every time, but the message is different for each combination. Some show Data must start with zero, while others are Message is larger than modulus.

The iOS: PKCS1, Java: RSA doesn't throw an exception, but the resulting decrypted byte[] array should be length 16, but it's length 256, which means the padding isn't correctly stripped out.

Can someone help?

*** EDIT ***

As I'm doing more testing, I came across this page (http://javadoc.iaik.tugraz.at/iaik_jce/current/iaik/pkcs/pkcs1/RSACipher.html), which essentially tells me that RSA == RSA/None/PKCS1Padding. The decryption works in the sense that there are no exceptions, but I'm still getting a decrypted key whose byte[] is length 256 instead of length 16.

Another point of interest. It seems that if the Java server has the public key generated from the iOS device and encrypted using Cipher.getInstance("RSA"), the phone is able to decode the message correctly with RSA/PKCS1.

*** EDIT 2 ***

I have looked at these tutorials and looked through my code again on the iOS side:

• http://blog.flirble.org/2011/01/05/rsa-public-key-openssl-ios/
• http://blog.wingsofhermes.org/?p=42
• http://blog.wingsofhermes.org/?p=75

As far as I can tell, my code is doing everything correctly. One significant difference was in how I was saving the key, so I tried saving it the other way:

    OSStatus error = noErr;
    CFTypeRef persistPeer = NULL;

    NSMutableDictionary * keyAttr = [[NSMutableDictionary alloc] init];

    keyAttr[(__bridge id) kSecClass] = (__bridge id) kSecClassKey;
    keyAttr[(__bridge id) kSecAttrKeyType] = (__bridge id) kSecAttrKeyTypeRSA;
    keyAttr[(__bridge id) kSecAttrApplicationTag] = [secKeyWrapper getKeyTag:serverPublicKeyTag];
    keyAttr[(__bridge id) kSecValueData] = strippedServerPublicKey;
    keyAttr[(__bridge id) kSecReturnPersistentRef] = @YES;

    error = SecItemAdd((__bridge CFDictionaryRef) keyAttr, (CFTypeRef *)&persistPeer);

    if (persistPeer == nil || ( error != noErr && error != errSecDuplicateItem)) {
        NSLog(@"Problem adding public key to keychain");
        return;
    }

    CFRelease(persistPeer);

That save was successful, but the end result was the same: the decrypted AES key was still 256 bytes long instead of 16 bytes.

Answer 1

I had same problem. Does work with kSecPaddingNone, but doesn't work with kSecPaddingPKCS1 with any PKCS1 combination in Java code.

But, it's not good idea to use it without padding.

So, on iOS, replace kSecPaddingNone with kSecPaddingOAEP and use RSA/NONE/OAEPWithSHA1AndMGF1Padding in your Java code. This does work for me.