avformat/rm: limit packet size

The chunk size is limited to 0xFFFF (written by avio_wb16), so make
sure that the packet size is not too large.

Such large frames need to be split into slices smaller than 64 kB, but
that is currently supported neither by the rv10/rv20 encoders nor the rm
muxer.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
See Ticket244

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 08728f400b8367dc8c983036cb2eff3a2891322b)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

diff --git a/libavformat/rmenc.c b/libavformat/rmenc.c
index 17192ff..a4e97a0 100644 (file)
--- a/libavformat/rmenc.c
+++ b/libavformat/rmenc.c
@@ -391,6 +391,11 @@ static int rm_write_video(AVFormatContext *s, const uint8_t *buf, int size, int
     /* Well, I spent some time finding the meaning of these bits. I am
        not sure I understood everything, but it works !! */
 #if 1
+    /* 0xFFFF is the maximal chunk size; header needs at most 7 + 4 + 12 B */
+    if (size > 0xFFFF - 7 - 4 - 12) {
+        av_log(s, AV_LOG_ERROR, "large packet size %d not supported\n", size);
+        return AVERROR_PATCHWELCOME;
+    }
     write_packet_header(s, stream, size + 7 + (size >= 0x4000)*4, key_frame);
     /* bit 7: '1' if final packet of a frame converted in several packets */
     avio_w8(pb, 0x81);