avformat/vqf: Use 64bit for ret to avoid overflow

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit cb08687180683a755d0fe9d425280d0e4d1e6db2)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/mvdec: Use 64bit for ret to avoid overflow

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 26c0cc154e06cb0064b3a3da49447ac44d82444f)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/gxf: Use 64bit for res to avoid overflow

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 12987f89007ee82b9d3a6090085dfaef8461ab8b)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/idcin: Use 64bit for ret to avoid overflow

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit d1923d15a3544cbb94563a59e7169291db76b312)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/h264_slice: ignore SAR changes in slices after the first

Fixes race condition and null pointer dereference
Fixes: signal_sigsegv_1472ac3_468_cov_2915641226_CABACI3_Sony_B.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 38d5241b7f36c1571a88517a0650caade16dd5f4)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Conflicts:

libavcodec/h264_slice.c

avcodec/h264_slice: Check picture structure before setting the related fields

This might fix a hypothetical race condition

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f111831ed61103f9fa8fdda41473a23da016bdaa)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Conflicts:

libavcodec/h264_slice.c

avcodec/h264_slice: Do not change frame_num after the first slice

Fixes potential race condition
Fixes: signal_sigsegv_1472ac3_468_cov_2915641226_CABACI3_Sony_B.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f906982c9411f3062e3ce68013309b37c213c4dd)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Conflicts:

libavcodec/h264_slice.c

avcodec/h264: Be more strict on rejecting pps/sps changes

Fixes race condition
Fixes: signal_sigsegv_1472ac3_468_cov_2915641226_CABACI3_Sony_B.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 6fafc62b0bd0e206deb77a7aabbf3a370ad80789)

Conflicts:

libavcodec/h264.c

avcodec/h264_ps: More completely check the bit depths

Fixes out of array read
Fixes: asan_static-oob_30328b6_719_cov_3325483287_H264_artifacts_motion.h264

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 69aa79365c1e8e1cb597d33e77bf1062c2ef47d4)

Conflicts:

libavcodec/h264_ps.c

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/thp: Check av_get_packet() for failure not only for partial output

Fixes null pointer dereference
Fixes: signal_sigsegv_db2c1f_3108_cov_163322880_pikmin2_opening1_partial.thp

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f2579dbb4b31e6ae731e7f5555680528ef3020ab)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

swscale/utils: Limit filter shifting so as not to read from prior the array

Fixes out of array read
Fixes: asan_heap-oob_1fb2f9b_3780_cov_3984375136_usf.mkv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 692b22626ec9a9585f667c124a186b1a9796e432)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/mjpegdec: Check escape sequence validity

Fixes assertion failure
Fixes: asan_heap-oob_1c1a4ea_1242_cov_2274415971_TESTcmyk.jpg

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/mjpegdec: Check number of components for JPEG-LS

Fixes out of array accesses
Fixes: asan_heap-oob_1c1a4ea_1242_cov_2274415971_TESTcmyk.jpg

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit fabbfaa095660982cc0bc63242c459561fa37037)

Conflicts:

libavcodec/mjpegdec.c

avformat/mpc8: Use uint64_t in *_get_v() to avoid undefined behavior

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 05e161952954acf247e0fd1fdef00559675c4d4d)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/mpc8: fix broken pointer math

This could overflow and crash at least on 32 bit systems.

Reviewed-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit b737a2c52857b214be246ff615c6293730033cfa)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/mpc8: fix hang with fuzzed file

This can lead to an endless loop by seeking back a few bytes after each
attempted chunk read. Assuming negative sizes are always invalid, this
is easy to fix. Other code in this demuxer treats negative sizes as
invalid as well.

Fixes ticket #4262.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 56cc024220886927350cfc26ee695062ca7ecaf4)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/tta: fix crash with corrupted files

av_add_index_entry() can fail, for example because the parameters are
invalid, or because memory allocation fails. Check this; it can actually
happen with corrupted files.

The second hunk is just for robustness. Just in case functions like
ff_reduce_index() remove entries. (Not sure if this can actually
happen.)

Fixes ticket #4294.

Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 6a0cd529a35190d9374b0b26504e71857cd67b83)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/mpegvideo_enc: Fix number suffixes in rc_buffer_size calculation

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 4531e2c489d279bfc90d54ca26ed898c5b265a7f)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/h264_cabac: use int instead of long for mbb_xy

The mb address fits in int

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 592ba6ec106206f97133c9345313010c76361e12)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/omadec: fix number suffix

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f1f7f5903ab49b84789af5341492afbaba808a70)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/smacker: Fix number suffix

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 465f3705b1ef832fd6904750d018f81f9044f3ab)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/matroskadec: Fix number suffixes

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit fc3cdb00d084222a107e61e7168903bf3d3d0b47)

Conflicts:

libavformat/matroskadec.c

swresample/dither: Cleanup number suffixes

The <<31 case needs LL

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c77cc2c1766666cdb5f14daee0f75e397bf7a194)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/utils: Fix number suffixes in tb_unreliable()

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 4b15bba2aec93776bfdc69a1bca42a4795a7d191)

Conflicts:

libavformat/utils.c

avformat/rmdec: Check for overflow in ff_rm_read_mdpr_codecdata()

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 03abf55f252945c70f4a79eaf4d609cee4d98710)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

swscale: check memory allocations

Bug-Id: CID 1257779
(cherry picked from commit 1dd797e3c9f179f957316a0becbec048b42df8aa)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/flac_parser: fix handling EOF if no headers are found

Fixes assertion failure
Fixes Ticket4269

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c4d85fc23c100f7a27d9bad710eb153214868e27)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

ffmpeg: Clear error message array at init.

This avoids printing uninitialized bytes if no error message is set

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 6d1a2efb8ac399a003ea7d3b6f8c641d192567ee)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/dvdsubdec: fix accessing dangling pointers

dvdsub_decode() can call append_to_cached_buf() 2 times, the second time
with ctx->buf as argument. If the second append_to_cached_buf() reallocs
ctx->buf, the argument will be a pointer to the previous, freed block.
This can cause invalid reads at least with some fuzzed files - and
possibly with valid files.

Since packets can apparently not be larger than 64K (even if packets are
combined), just use a fixed size buffer. It will be allocated as part of
the DVDSubContext, and although some memory is "wasted", it's relatively
minimal by modern standards and should be acceptable.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 816577716bc6170bccfea3b9e865618b69a4b426)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/dvdsubdec: error on bitmaps with size 0

Attemtping to decode them could lead to invalid writes with some fuzzed
samples.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit bcaa9099b3648b47060e1724a97dc98b63c83702)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

cmdutils: update copyright year to 2015.

(cherry picked from commit 3e160652219ff4da433f5672ae1e5f4956abb815)

Conflicts:

cmdutils.c

avformat/mov: Fix mixed declaration and statement warning

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit db27f50e0658e91758e8a17fdcf390e6bc93c1d2)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/utils: Clear pointer in ff_alloc_extradata() to avoid leaving a stale pointer in memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit bbfca8e84b0e69abba523d665536c0135fc1c00e)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/matroskadec: Use av_freep() to avoid leaving stale pointers in memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 6e70e4aca50696040cc9256ec96e5c31d9641432)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

lavfi: check av_strdup() return value

Signed-off-by: Paul B Mahol <onemda@gmail.com>
(cherry picked from commit 145a84717b62e086cdb5f26649ad9f1b51ef38d0)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

mov: Fix negative size calculation in mov_read_default().

The previous code assumed if an atom was marked with a 64-bit
size extension, it actually had that data available. The new
code verfies there's enough data in the atom for this to be
done.

Failure to verify causes total_size > atom.size which will
result in negative size calculations later on.

Found-by: Paul Mehta <paul@paulmehta.com>
Signed-off-by: Dale Curtis <dalecurtis@chromium.org>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3ebd76a9c57558e284e94da367dd23b435e6a6d0)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/mov: fix integer overflow in mov_read_udta_string()

Found-by: Paul Mehta <paul@paulmehta.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3859868c75313e318ebc5d0d33baada62d45dd75)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

mov: Avoid overflow with mov_metadata_raw()

The code previously added 1 to len without checking its size,
resulting in an overflow which can corrupt value[-1] -- which
may be used to store unaligned ptr information for certain
allocators.

Found-by: Paul Mehta <paul@paulmehta.com>
Signed-off-by: Dale Curtis <dalecurtis@chromium.org>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/dvdsubdec: fix out of bounds accesses

The code blindly trusted buffer offsets read from the file in the RLE
decoder. Explicitly check the offset. Also error out on other RLE
decoding errors.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c9151de7c42553bb145be608df8513c1287f1f24)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avfilter/vf_sab: fix filtering tiny images

Fixes out of array reads

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 9bff052b51f27f6cce04e8d7d8b405c710d7ad67)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/flvdec: Increase string array size

Fixes parsing httphostheader of Scarlatti\,\ Pieter-Jan\ Belder\ -\ Sonata\ K113\ in\ A\ major\ -\ Alle.flv

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit eb767a276bfdb9a0493bdb0b38203638230b7ccb)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/flvdec: do not inject dts=0 metadata packets which failed to be parsed into a new data stream

Such data streams (which then contain no other packets except the faulty one)
confuse some user applications, like VLC
Works around vlcticket 12389

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 322f0f5960a743cac47252d90a0f1ea7a025feff)

Conflicts:

libavformat/flvdec.c

avformat/cdxl: Fix integer overflow of image_size

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3eb5cbe0c50d0a0bbe10bcabbd6b16d73d93c128)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Update for 2.1.7

lavf/segment: remove duplicated and inconsistent cleanup code in seg_write_packet()

In particular, avoid to leave around the seg->avf pointer to freed
structure, and fix crash with:
ffmpeg -f lavfi -i testsrc -c:v h264 -map 0 -f segment foo-%d.ts
(cherry picked from commit 169065fbfb3da1ab776379c333aebc54bb1f1bc4)

Found-by: Qinghao Tang
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/mov: Fix memleaks for duplicate STCO/CO64/STSC atoms

Also see [FFmpeg-devel] [PATCH] avformat/mov: strengthen some table allocations
which contains more fixes but is unfinished

Fixes: signal_sigabrt_7ffff6ac7bb9_3484_cov_1830000177_starfox2.mov
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 1b5d11240692025f036e945bc37968735679320a)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

mmvideo: check frame dimensions

The frame size must be set by the caller and each dimension must be a
multiple of 2.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
See: 8b0e96e1f21b761ca15dbb470cd619a1ebf86c3e
These should be redundant, but are backported for saftey anyway
(cherry picked from commit b0273232d8fffdc8a977ccdad460b8071a0e353c)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

jvdec: check frame dimensions

The frame size must be set by the caller and each dimension must be a
multiple of 8.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
See: 105654e376a736d243aef4a1d121abebce912e6b
These should be redundant, but are backported for saftey anyway
(cherry picked from commit e012cb8dea7969c7b3927dbf846ef2742cd4a7ab)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/indeo3: ensure offsets are non negative

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 368642361f3a589d7b0c23ea327d988edb434e3f)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/h264: Check *log2_weight_denom

Fixes undefined behavior
Fixes: signal_sigsegv_14768d2_2248_cov_3629497219_h264_h264___pi_20070614T182942.h264
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 61296d41e2de3b41304339e4631dd44c2e15f805)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/hevc_ps: Check diff_cu_qp_delta_depth

Fixes undefined behavior
Fixes: asan_static-oob_17aa046_582_cov_1577759978_DBLK_G_VIXS_1.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3281fa892599d71b4dc298a426af8296419cd90e)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/h264: Clear delayed_pic on deallocation

Fixes use of freed memory

Fixes: case5_av_frame_copy_props.mp4
Found-by: Michal Zalewski <lcamtuf@coredump.cx>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e8714f6f93d1a32f4e4655209960afcf4c185214)

Conflicts:

libavcodec/h264.c

avcodec/hevc: clear filter_slice_edges() on allocation

This avoids use of uninitialized memory
Fixes: asan_static-oob_17aa046_582_cov_212287884_DBLK_G_VIXS_1.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8aa8d12554868c32436750f881954193087219c8)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/indeo3: use signed variables to avoid underflow

Fixes out of array read
Fixes: signal_sigsegv_1b0a4da_1865_cov_2167818389_computer_anger.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3305acdc92fa37869f160a11a87741c8a0de0454)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/h264: make the first field of H264Context an AVClass

Fixes use of freed memory
Fixes: asan_heap-uaf_3660f67_757_cov_1257014655_Hi422FR1_SONY_A.jsv
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f3b5b139ad853b6f69c6a0b036815a60e7b3f261)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

swscale: increase yuv2rgb table headroom

Fixes out of array access
Fixes: case2_bad_read_yuv2rgbx32.mp4
Found-by: Michal Zalewski <lcamtuf@coredump.cx>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/mov: check atom nesting depth

Fixes call stack overflow
Fixes: case1_call_stack_overflow.mp4
Found-by: Michal Zalewski <lcamtuf@coredump.cx>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit caa7a3914f499f74b3ee346f26d598ebdc0ec210)

Conflicts:

libavformat/isom.h

Conflicts:

libavformat/isom.h

avcodec/utvideodec: Fix handling of slice_height=0

Fixes out of array accesses
Fixes: asan_heap-oob_25bcd7e_3783_cov_3553517262_utvideo_rgba_median.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3881606240953b9275a247a1c98a567f3c44890f)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/vmdvideo: Check len before using it in method 3

Fixes out of array access
Fixes: asan_heap-oob_4d23ba_91_cov_3853393937_128.vmd

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3030fb7e0d41836f8add6399e9a7c7b740b48bfd)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

configure: create the tests directory like the doc directory

This fixes an issue where the tests directory is not created for out of tree
builds before its needed

Tested-by: Dave Yeo <daveryeo@telus.net>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e631872f13b6be0583603d45a11e53319754bc8d)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

lavu/frame: fix malloc error path in av_frame_copy_props()

The error path frees all side data, but forgets to reset the side data
count. This can blow up later in av_frame_unref() and free_side_data().

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit a400edbb6d00c0211de38e4f1b4f593681db91d8)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/aviobuf: Check that avio_seek() target is non negative

Fixes out of array access

Suggested-by: Andrew Scherkus <scherkus@google.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit ed86dbd05d61363dc1c0d33f3267e2177c985fdd)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

swresample/soxr_resample: fix error handling

Fixes CID1257659

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 4b6f2253741f3023928e61ae5105ccd4b1c515fb)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/matroskadec: fix handling of recursive SeekHead elements

When matroska_execute_seekhead() is called, it goes through the list of
seekhead entries and attempts to read elements not read yet. When doing
this, the parser can find further SeekHead elements, and will extend the
matroska->seekhead list. This can lead to a (practically) infinite loop
with certain broken files. (Maybe it can happen even with valid files.
The demuxer doesn't seem to check correctly whether an element has
already been read.)

Fix this by ignoring elements that were added to the seekhead field
during executing seekhead entries.

This does not fix the possible situation when multiple SeekHead elements
after the file header (i.e. occur after the "before_pos" file position)
point to the same elements. These elements will probably be parsed
multiple times, likely leading to bugs.

Fixes ticket #4162.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 6551acab6877addae815decd02aeca33ba4990c8)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

swscale/x86/rgb2rgb_template: fix crash with tiny size and nv12 output

Fixes Ticket4151

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8524558858b7e14bc50afa10233e0194f591ab9d)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/rmdec: Check codec_data_size

Fixes infinite loop
Fixes Ticket4154

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit a6f730730b82645a9d31aad0968487cb77d6946c)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/motion_est: use 2x8x8 for interlaced qpel

Fixes out of array read
Fixes Ticket4121

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit b50e003e1cb6a215df44ffa3354603bf600b4aa3)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

fix Makefile objects for pulseaudio support

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit cb58c771ade66afcc623250e1c7ac8191381d991)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/rsd: make tag_buf string larger

av_get_codec_tag_string() uses more that 1 char for unprintable characters.

(cherry picked from commit edbbb11488e1fce9b9703535936d2e1731e2e318)

Update for 2.1.6

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/svq3: Dont memcpy AVFrame

This avoids out of array accesses

Fixes: asan_heap-uaf_21f42e4_9_asan_heap-uaf_21f42e4_278_gl2.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 075a165d2715837d125a9cc714fb430ccf6c9d6b)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/hevc_ps: Check num_long_term_ref_pics_sps

Fixes out of array access
Fixes: signal_sigsegv_35bd0f0_1182_cov_791726764_STRUCT_B_Samsung_4.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit ea38e5a6b75706477898eb1e6582d667dbb9946c)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/rawdec: Check the return code of avpicture_get_size()

Fixes out of array access
Fixes: asan_heap-oob_22388d0_3435_cov_3297128910_small_roll5_FlashCine1.cine
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 1d3a3b9f8907625b361420d48fe05716859620ff)

Conflicts:

libavcodec/rawdec.c

avcodec/pngdec: Check IHDR/IDAT order

Fixes out of array access
Fixes: asan_heap-oob_20a6c26_2690_cov_3434532168_mail.png
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 79ceaf827be0b070675d4cd0a55c3386542defd8)

Conflicts:

libavcodec/pngdec.c

avcodec/flacdec: Call ff_flacdsp_init() unconditionally

Fixes out of array access
Fixes: signal_sigsegv_324b135_3398_cov_246853371_short.flac
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e5c01ccdf5a9a330d4c51a9b9ea721fd8f1fb70b)

Conflicts:

libavcodec/flacdec.c

avcodec/utils: Check that the data is complete in avpriv_bprint_to_extradata()

Fixes out of array read
Fixes: asan_heap-oob_4d2250_814_cov_2745172097_JACOsub_capability_tester.jss
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3d5d95db3f5d8e2093e9e19d0c46e86f54ed2a5d)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/mjpegdec: Fix context fields becoming inconsistent

Fixes out of array access
Fixes: asan_heap-oob_1ca4f85_2760_cov_144449187_miss_congeniality_pegasus_ljpg.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 0eecf40935b22644e6cd74c586057237ecfd6844)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

swscale/x86/rgb2rgb_template: handle the first 2 lines with C in rgb24toyv12_*()

This avoids out of array accesses
Should fix Ticket3451

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 4388e78a0f022c8572996f9ab568a39b5f716f9d)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/hlsenc: Free context after hls_append_segment

Fixes reading uninitialized memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 530eb6acf8ee867bf00728bf7efaf505da107e17)

Conflicts:

libavformat/hlsenc.c
(cherry picked from commit 0ac22f043bee2f1c4daf5e1044b014326325d929)

Conflicts:

libavformat/hlsenc.c
(cherry picked from commit 134d3e1c0331462ea94c78a5e13a63b20d283653)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/wmaprodec: Fix integer overflow in sfb_offsets initialization

Fixes out of array read
Fixes: asan_heap-oob_2aec5b0_1828_classical_22_16_2_16000_v3c_0_exclusive_0_29.wma
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5dcb99033df16eccc4dbbc4a099ad64457f9f090)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/utvideodec: fix assumtation that slice_height >= 1

Fixes out of array read
Fixes: asan_heap-oob_2573085_3783_utvideo_rgba_median.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 7656c4c6e66f8a787d384f027ad824cc1677fda1)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/options_table fix min of audio channels and sample rate

Found-by: Lukasz Marek <lukasz.m.luki2@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 206c98f303e833c9e94427c9e3f9867f85265f78)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

lavc/utils: Make pix_fmt desc pointer const.

Fixes an "initialization discards qualifiers from pointer target type" warning.
(cherry picked from commit f05855414ed4cce97c06ba2a31f4987af47e6d4e)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/h264_slice: Clear table pointers to avoid stale pointers

Might fix Ticket3889

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 547fce95858ef83f8c25ae347e3ae3b8ba437fd9)

Conflicts:

libavcodec/h264_slice.c

avcodec/svq1dec: zero terminate embedded message before printing

Fixes out of array access
Fixes: asan_stack-oob_49b1e5_10_009.mov
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e91ba2efa949470e9157b652535d207a101f91e0)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/cook: check that the subpacket sizes fit in block_align

Fixes out of array read
Fixes: asan_heap-oob_fb5c50_19_018.rmvb
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 10e32618acce9c3fc64c061eb7907e8a8d2749ae)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/g2meet: check tile dimensions to avoid integer overflow

Fixes out of array access
Fixes: asan_heap-oob_12a55d3_30_029.wmv
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 32e666c354e4a3160d8cf1d303cb51990b095c87)

Conflicts:

libavcodec/g2meet.c

avcodec/utils: Align dimensions by at least their chroma sub-sampling factors.

Fixes: out of array accesses
Fixes: asan_heap-oob_112c6b3_13_012.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit df74811cd53e45fcbbd3b77a1c42416816687c5c)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/dnxhddec: treat pix_fmt like width/height

Fixes out of array accesses
Fixes: asan_heap-oob_22c9a39_16_015.mxf
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f3c0e0bf6f53df0977f3878d4f5cec99dff8de9e)

Conflicts:

libavcodec/dnxhddec.c

avcodec/dxa: check dimensions

Fixes out of array access
Fixes: asan_heap-oob_11222fb_21_020.dxa
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e70312dfc22c4e54d5716f28f28db8f99c74cc90)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/dirac_arith: fix integer overflow

Fixes: asan_heap-oob_1078676_9_008.drc
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 39680caceebfc6abf09b17032048752c014e57a8)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/diracdec: Tighter checks on CODEBLOCKS_X/Y

Fixes very long but finite loop
Fixes: asan_heap-oob_107866c_42_041.drc
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5145d22b88b9835db81c4d286b931a78e08ab76a)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/diracdec: Use 64bit in calculation of codeblock coordinates

Fixes integer overflow
Fixes out of array read
Fixes: asan_heap-oob_107866c_42_041.drc
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 526886e6069636a918c8c04db17e864e3d8151c1)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

postproc: fix qp count

Found-by: ubitux
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 0b7e5d0d75e7d8762dd04d35f8c0821736164372)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

postproc/postprocess: fix quant store for fq mode

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 941aaa39e8cd78ba4d16dfcec767290aec9a0136)

Conflicts:

tests/ref/fate/filter-pp3
(cherry picked from commit 705748caf3f6a4a3e74ad3d2fc547a5a0213a521)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

utvideoenc: properly set slice height/last line

Mimic decoder and obey sampling.

Does not affect fate tests for utvideo.
Fixes ticket #3949.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit cb530dda7d76790b08ee3b7f67e251f3ce48c359)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

swresample/swresample: fix sample drop loop end condition

Fixes Ticket3985

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f9fefa499f0af48f47ea73c8ce0b25df0976c315)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/h264: Check mode before considering mixed mode intra prediction

Fixes out of array read
Fixes: asan_heap-oob_e476fc_2_asan_heap-oob_1333ec6_61_CAMACI3_Sony_C.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 9734a7a1de3043f012ad0f1ef11027d9488067e6)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/mpegts: Check desc_len / get8() return code

Fixes out of array read
Fixes: signal_sigsegv_844d59_10_signal_sigsegv_a17bb7_366_mpegts_mpeg2video_mp2_dvbsub_topfield.rec

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c3d7f00ee3e09801f56f25db8b5961f25e842bd2)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/vorbisdec: Fix off by 1 error in ptns_to_read

Fixes read of uninitialized memory
Fixes: asan_heap-uaf_18dac2b_9_asan_heap-uaf_22eb375_208_beta3_test_small.ogg

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8c50704ebf1777bee76772c4835d9760b3721057)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>