Fix YV12 dumb buffer plane sizes

Allocating a 1280x720 YUV420 (YV12) buffer results in
calling the create dumb ioctl with 1280x1080 and results
in a file with size 1384448
(1280 * 1080 = 1382400, ALIGN(1382400, 4096) = 1384448).

The existing code calls drv_bo_from_format with the
aligned height of 768 which results in oversized planes
totaling a size of 1474560 bytes. This ultimately leads
to segfaults when running past the end of the mapped
file.

Because the dumb buffer iotcl height is computed using
the buffer's unaligned height (bo->meta.height), the
call to drv_bo_from_format should use the unaligned
height as well.

BUG=b:146515640
TEST=cts android.media.cts.EncodeDecodeTest

Change-Id: I40893cf92caa9d35a89d21a3512604f361822222
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/minigbm/+/2171157
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>
Tested-by: Jason Macnak <natsu@google.com>
Commit-Queue: Jason Macnak <natsu@google.com>

diff --git a/helpers.c b/helpers.c
index 22a6106..fed4af9 100644 (file)
--- a/helpers.c
+++ b/helpers.c
@@ -313,14 +313,17 @@ int drv_dumb_bo_create_ex(struct bo *bo, uint32_t width, uint32_t height, uint32
        aligned_height = height;
        switch (format) {
        case DRM_FORMAT_YVU420_ANDROID:
+               /* HAL_PIXEL_FORMAT_YV12 requires that the buffer's height not
+                * be aligned. Update 'height' so that drv_bo_from_format below
+                * uses the non-aligned height. */
+               height = bo->meta.height;
+
                /* Align width to 32 pixels, so chroma strides are 16 bytes as
                 * Android requires. */
                aligned_width = ALIGN(width, 32);
-               /* Adjust the height to include room for chroma planes.
-                *
-                * HAL_PIXEL_FORMAT_YV12 requires that the buffer's height not
-                * be aligned. */
-               aligned_height = 3 * DIV_ROUND_UP(bo->meta.height, 2);
+
+               /* Adjust the height to include room for chroma planes. */
+               aligned_height = 3 * DIV_ROUND_UP(height, 2);
                break;
        case DRM_FORMAT_YVU420:
        case DRM_FORMAT_NV12: