The SharedPointer::alloc implementation for sync::Arc<T> and rc::Rc<T> in rkyv/src/impls/alloc/rc/atomic.rs (and rc.rs) does not check if the allocator returns a null pointer on OOM (Out of Memory).

This null pointer can flow through to SharedPointer::from_value, which calls Box::from_raw(ptr) with the null pointer. This triggers undefined behavior when utilizing safe deserialization APIs (such as rkyv::from_bytes or rkyv::deserialize_using) if an OOM condition occurs during the allocation of the shared pointer.

The issue is reachable through safe code and violates Rust's safety guarantees.

In the unique reclaim path of BytesMut::reserve, the condition

if v_capacity >= new_cap + offset

uses an unchecked addition. When new_cap + offset overflows usize in release builds, this condition may incorrectly pass, causing self.cap to be set to a value that exceeds the actual allocated capacity. Subsequent APIs such as spare_capacity_mut() then trust this corrupted cap value and may create out-of-bounds slices, leading to UB.

This behavior is observable in release builds (integer overflow wraps), whereas debug builds panic due to overflow checks.

PoC

use bytes::*;

fn main() {
    let mut a = BytesMut::from(&b"hello world"[..]);
    let mut b = a.split_off(5);

    // Ensure b becomes the unique owner of the backing storage
    drop(a);

    // Trigger overflow in new_cap + offset inside reserve
    b.reserve(usize::MAX - 6);

    // This call relies on the corrupted cap and may cause UB & HBO
    b.put_u8(b'h');
}

Workarounds

Users of BytesMut::reserve are only affected if integer overflow checks are configured to wrap. When integer overflow is configured to panic, this issue does not apply.