Summary

In the AES GCM implementation of decrypt_in_place_detached, the decrypted ciphertext (i.e. the correct plaintext) is exposed even if tag verification fails.

Impact

If a program using the aes-gcm crate's decrypt_in_place* APIs accesses the buffer after decryption failure, it will contain a decryption of an unauthenticated input. Depending on the specific nature of the program this may enable Chosen Ciphertext Attacks (CCAs) which can cause a catastrophic breakage of the cipher including full plaintext recovery.

Details

As seen in the implementation of decrypt_in_place_detached for AES GCM, if the tag verification fails, an error is returned. Because the decryption of the ciphertext is done in place, the plaintext contents are now exposed via buffer.

This should ideally not be the case - as noted in page 17 of NIST's publication Recommendation for Block Cipher Modes of Operation: Galois/Counter Mode (GCM) and GMAC:

In Step 8, the result of Step 7 is compared with the authentication tag that was received as an input: if they are identical, then the plaintext is returned; otherwise,FAIL is returned.

This is seems correctly addressed in the AES GCM SIV implementation, where the decrypted buffer is encrypted again before the error is returned - this fix is straightforward to implement in AES GCM. To ensure that these types of cases are covered during testing, it would be valuable to add test cases like 23, 24 etc from project wycheproof to ensure that when a bad tag is used, there is an error on decryption and that the plaintext value is not exposed.

PoC

To reproduce this issue, I'm using test case 23 from project wycheproof.

    let key = GenericArray::from_slice(&hex!("000102030405060708090a0b0c0d0e0f"));
    let nonce = GenericArray::from_slice(&hex!("505152535455565758595a5b"));
    let tag = GenericArray::from_slice(&hex!("d9847dbc326a06e988c77ad3863e6083")); // bad tag
    let mut ct = hex!("eb156d081ed6b6b55f4612f021d87b39");
    let msg = hex!("202122232425262728292a2b2c2d2e2f");
    let aad = hex!("");
    let cipher = Aes128Gcm::new(&key);
    let _plaintext = cipher.decrypt_in_place_detached(&nonce, &aad, &mut ct, &tag);
    assert_eq!(ct, msg);

In the unique reclaim path of BytesMut::reserve, the condition

if v_capacity >= new_cap + offset

uses an unchecked addition. When new_cap + offset overflows usize in release builds, this condition may incorrectly pass, causing self.cap to be set to a value that exceeds the actual allocated capacity. Subsequent APIs such as spare_capacity_mut() then trust this corrupted cap value and may create out-of-bounds slices, leading to UB.

This behavior is observable in release builds (integer overflow wraps), whereas debug builds panic due to overflow checks.

PoC

use bytes::*;

fn main() {
    let mut a = BytesMut::from(&b"hello world"[..]);
    let mut b = a.split_off(5);

    // Ensure b becomes the unique owner of the backing storage
    drop(a);

    // Trigger overflow in new_cap + offset inside reserve
    b.reserve(usize::MAX - 6);

    // This call relies on the corrupted cap and may cause UB & HBO
    b.put_u8(b'h');
}

Workarounds

Users of BytesMut::reserve are only affected if integer overflow checks are configured to wrap. When integer overflow is configured to panic, this issue does not apply.