linux-user/mmap.c: fix integer underflow in target_mremap

Fixes: https://bugs.launchpad.net/bugs/1876373

This code path in mmap occurs when a page size is decreased with mremap.  When a section of pages is shrunk, qemu calls mmap_reserve on the pages that were released.  However, it has the diff operation reversed, subtracting the larger old_size from the smaller new_size.  Instead, it should be subtracting the smaller new_size from the larger old_size.  You can also see in the previous line of the change that this mmap_reserve call only occurs when old_size > new_size.

Bug: https://bugs.launchpad.net/qemu/+bug/1876373
Signed-off-by: Jonathan Marler <johnnymarler@gmail.com>
Reviewded-by: Laurent Vivier <laurent@vivier.eu>
Message-Id: <20200502161225.14346-1-johnnymarler@gmail.com>
Signed-off-by: Laurent Vivier <laurent@vivier.eu>

diff --git a/linux-user/mmap.c b/linux-user/mmap.c
index e378033..caab629 100644 (file)
--- a/linux-user/mmap.c
+++ b/linux-user/mmap.c
@@ -708,7 +708,7 @@ abi_long target_mremap(abi_ulong old_addr, abi_ulong old_size,
         if (prot == 0) {
             host_addr = mremap(g2h(old_addr), old_size, new_size, flags);
             if (host_addr != MAP_FAILED && reserved_va && old_size > new_size) {
-                mmap_reserve(old_addr + old_size, new_size - old_size);
+                mmap_reserve(old_addr + old_size, old_size - new_size);
             }
         } else {
             errno = ENOMEM;