asfdec: reject size > INT64_MAX in asf_read_unknown

Both avio_skip and detect_unknown_subobject use int64_t for the size
parameter.

This fixes a segmentation fault due to infinite recursion.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Signed-off-by: Alexandra Hájková <alexandra.khirnova@gmail.com>
Signed-off-by: Anton Khirnov <anton@khirnov.net>

diff --git a/libavformat/asfdec.c b/libavformat/asfdec.c
index 58480dc..4fc0e3d 100644 (file)
--- a/libavformat/asfdec.c
+++ b/libavformat/asfdec.c
@@ -178,6 +178,9 @@ static int asf_read_unknown(AVFormatContext *s, const GUIDParseTable *g)
     uint64_t size   = avio_rl64(pb);
     int ret;
 
+    if (size > INT64_MAX)
+        return AVERROR_INVALIDDATA;
+
     if (asf->is_header)
         asf->unknown_size = size;
     asf->is_header = 0;