avformat/utils/av_probe_input_buffer2: fix offset check

The check could fail if avio_read() read less than requested

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/utils/av_probe_input_buffer2: Fix pd.buf_size

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Merge remote-tracking branch 'qatar/master'

* qatar/master:
  lavf: make av_probe_input_buffer more robust

Conflicts:
libavformat/utils.c

Not merged as it depends on previous buggy changes

See: e74cd2f4706f71da5e9205003c1d8263b54ed3fb
See: ececbedaa86cddd883f6cdc12cdd1a75a0023550
Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '8b76362836f3c373c3aadc544522edcbef16dd5f'

* commit '8b76362836f3c373c3aadc544522edcbef16dd5f':
  lavf: use a fixed width type

Conflicts:
libavformat/utils.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit 'c1868e7ee7b07b40a0fe15f50df89fe499a01a50'

* commit 'c1868e7ee7b07b40a0fe15f50df89fe499a01a50':
  lavf: simplify handling of offset in av_probe_input_buuffer()

Conflicts:
libavformat/utils.c

The simplification is broken and does not work at least for ffmpeg,
thus this change is not merged.

Merged-by: Michael Niedermayer <michaelni@gmx.at>

lavf: make av_probe_input_buffer more robust

Always use the actually read size as the offset instead of making
possibly invalid assumptions.

Addresses: CVE-2012-6618

lavf: use a fixed width type

It's shorter and more consistent with the rest of the code.

lavf: simplify handling of offset in av_probe_input_buuffer()

Merge commit '99e20125232d40ea239a5b54d26bbfe526506114'

* commit '99e20125232d40ea239a5b54d26bbfe526506114':
  x86/arm: Add clobber tests to libavresample

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '49b9badcaab513b65b79be371b1a9c1012e482ea'

* commit '49b9badcaab513b65b79be371b1a9c1012e482ea':
  vc1: Enable the interlaced B-frame codepath.

See: 33f2a4942380184f3a28cbf2a36366c8ed105232
Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '6d98959c8a5a749592a878b9523b31eccea5d102'

* commit '6d98959c8a5a749592a878b9523b31eccea5d102':
  vc1: Add avg_no_rnd_vc1_chroma_mc4_c()

Conflicts:
libavcodec/vc1dsp.c

See: dd6e291e4083b7af53968c0ae06330841f63405b
Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit 'c798a6fedc3dd9ed3ac1fb2d95819af58705a87e'

* commit 'c798a6fedc3dd9ed3ac1fb2d95819af58705a87e':
  vc1: Factorize out chroma MC

Merged-by: Michael Niedermayer <michaelni@gmx.at>

x86/arm: Add clobber tests to libavresample

Signed-off-by: Martin Storsjö <martin@martin.st>

vc1: Enable the interlaced B-frame codepath.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

vc1: Add avg_no_rnd_vc1_chroma_mc4_c()

Needed for proper interlaced support.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

vc1: Factorize out chroma MC

avcodec/hevc: Check entry point arrays for malloc failure

Fixes null pointer dereference
Fixes: signal_sigsegv_e1d3b6_2192_DBLK_F_VIXS_2.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

hevc: Bound check slice_qp

The T-REC-H.265-2013044 page 79 states they have to be into the range
[-s->sps->qp_bd_offset, 51].

Fixes: asan_stack-oob_eae8e3_9522_WP_MAIN10_B_Toshiba_3.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

hevc: Reject impossible dependent tile

The tile 0 cannot depend on a previous one.
Prevent an out of array bound load in ff_hevc_cabac_init().

Fixes: asan_heap-oob_e3a924_1630_DBLK_A_MAIN10_VIXS_2.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Reviewed-by: Guillaume Martres <smarter@ubuntu.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/huffman: raise the input capability of ff_huff_gen_len_table() from 8 to 14bit

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/huffman: extend ff_huff_gen_len_table() to allow >8bit

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

vp9/x86: add ff_vp9_loop_filter_[vh]_16_16_ssse3().

16662 decicycles in loop_filter_h_16_16_c, 8387355 runs, 1253 skips
17510 decicycles in loop_filter_v_16_16_c, 8387516 runs, 1092 skips

4941 decicycles in ff_vp9_loop_filter_h_16_16_ssse3, 8387887 runs, 721 skips
3899 decicycles in ff_vp9_loop_filter_v_16_16_ssse3, 8387980 runs, 628 skips

Overall decode time goes from:
  ./ffmpeg -v 0 -nostats -threads 1 -i ~/samples/vp9/ped1080p.webm -f null -  8.10s user 0.02s system 99% cpu 8.126 total
to:
  ./ffmpeg -v 0 -nostats -threads 1 -i ~/samples/vp9/ped1080p.webm -f null -  6.15s user 0.04s system 99% cpu 6.199 total

(46 to 61 fps)

vp9/x86: factor out some code in VP9_UNPACK_MULSUB_2W_4X.

vp9/x86: remove reg redundancy in VP9_MULSUB_2W_2X.

vp9/x86: merge IDCT coef macros.

Merge remote-tracking branch 'qatar/master'

* qatar/master:
  vc1dsp: K&R formatting cosmetics

Conflicts:
libavcodec/vc1dsp.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>

swscale: disable ARM code until its build failure with clang/iphone is fixed

See: "19:40 Yu Xiaolei Re: [FFmpeg-devel] [PATCH] fix build with gas-preprocessor.pl"

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

vc1dsp: K&R formatting cosmetics

Signed-off-by: Diego Biurrun <diego@biurrun.de>

Revert "swscale: disable ARM code until its build failure with clang/iphone is fixed"

This reverts commit c8c7736c1025bcf5bb27e104a0d0eae749408739.

fix build with gas-preprocessor.pl

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

mpegts muxer, DVB subtitles encoder: common DVB subtitles payload

Improved DVB subtitles encoder to generate AVPacket.data in the same
format as generates MPEGTS demuxer + DVB subtitles parser. So now single
format of DVB subtitles data is used across all the components of FFmpeg:
only subtitles payload WITHOUT 0x20 0x00 bytes at the beginning and 0xFF
trailing byte.

Improved MPEGTS muxer to support format of DVB subtitles in
AVPacket.data described above: while muxing we add two bytes 0x20 0x00 to
the beginning of and 0xFF to the end of DVB subtitles payload.

The patch fixes DVB subtitle copy problems: tickets #2989 fully and #2024
partly.

Signed-off-by: Clément Bœsch <u@pkh.me>

examples/muxing: simplify video PTS setting

Rely on frame_count. Also more consistent with audio path.

avcodec/dct-test: add support for C prores IDCT

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

hevc: remove useless clip in FUNC(sao_band_filter)()

The src buffer should only contain values in the interval
[0, (1 << BIT_DEPTH) - 1]. Since shift = (BIT_DEPTH - 5), src[x] >> shift
must be in the interval [0, 31], so no clip is needed.

This removes the code that was changed in 5856bca360c5bc3e340a357d91b1f993c80a7bea
as the clip that was repositioned in that commit is removed

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

hevc: clip pixels when transquant bypass is used

Fixes: asan_stack-oob_eae8e3_7333_WPP_B_ericsson_MAIN10_2.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
This is a more proper fix than 5856bca360c5bc3e340a357d91b1f993c80a7bea

The reconstructed picture should always be clipped (see section 8.6.5),
previously we did not clip coding units where
cu_transquant_bypass_flag == 1

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/dct-test: reproduce 4..1019 clipping when testing prores IDCT

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/dct-test: correct output bias of prores idct

This way the output values make sense

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

vp9: fix mt-related hang a parser infinite loop.

Fixes trac ticket 3274.

Looked-at-by: ubitux
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/mjpegenc: use a seperate chroma matrix when luma and chroma differ

drop hardcoded TWOMATRIX code

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/mpegvideo_enc: init chroma matrix for mjpeg

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/mjpegenc: pass chroma quantization matrix through as well, not just luma

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Merge remote-tracking branch 'qatar/master'

* qatar/master:
  flv: Workaround for buggy Omnia A/XE encoder

Conflicts:
libavformat/flvdec.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>

hevc: Clip the pixel before shifting

Prevent an out of array bound read.

Fixes: asan_stack-oob_eae8e3_7333_WPP_B_ericsson_MAIN10_2.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/hevc: use av_mallocz() for allocating tab_ipm

Fixes use of uninitialized memory and out of stack array read
Fixes: signal_sigsegv_ecc526_7846_WPP_C_ericsson_MAIN_2.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

flv: Workaround for buggy Omnia A/XE encoder

The Omnia A/XE encoder writes the explicit extra data incorrectly
and wrongly disables parametric stereo. Truncating the extra data
by setting the size to 2 works around this. The AAC extra data
parser will then only parse the correct parts.

Bug-id: 599

avcodec/alac: warn if not all channels where decoded

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/alac: only set *got_frame_ptr when all channels have been decoded

Fixes use of uninitialized memory
Fixes: msan_uninit-mem_7f8b64436530_7895_quicktime_newcodec_applelosslessaudiocodec.m4a
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Merge remote-tracking branch 'qatar/master'

* qatar/master:
  arm: Add an option for making sure NEON registers aren't clobbered

Conflicts:
configure

Merged-by: Michael Niedermayer <michaelni@gmx.at>

avformat/pjsdec: dont increase pointer when its already at the end in read_ts()

Fixes use of uninitialized memory
Fixes: msan_uninit-mem_7f91f2de7764_2649_PJS_capability_tester.pjs
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/wmalosslessdec: shrink output on error so no uninitialized data is returned

Fixes use of uninitialized memory
partly fixes: msan_uninit-mem_7f7834b6a530_6473_luckynight-partial.wma
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/wmalosslessdec: deallocate uninitialized frame on decode_tilehdr() failure

Fixes use of uninitialized memory
partly fixes: msan_uninit-mem_7f7834b6a530_6473_luckynight-partial.wma
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/wmalosslessdec: Pass on error code from decode_tilehdr()

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Merge remote-tracking branch 'cehoyos/master'

* cehoyos/master:
  Cast signed libopenjpeg data values to unsigned before shifting.

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Cast signed libopenjpeg data values to unsigned before shifting.

This avoids undefined behaviour on left-shift.

Reviewed-by: Michael Bradshaw

vdpau: restore compatibility with deprecated fields in AVVDPAUContext

Fixes ticket #3133.

Signed-off-by: Hendrik Leppkes <h.leppkes@gmail.com>
Tested-by: EricV
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/matroskadec: check generic audio deinterleaver sub_packet_size against frame_size

Fixes use of uninitialized memory
Fixes: msan_uninit-mem_7f67d052a530_7517_nosound.mkv
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

arm: Add an option for making sure NEON registers aren't clobbered

This is pretty much based on the same test for XMM registers.

Signed-off-by: Martin Storsjö <martin@martin.st>

Merge remote-tracking branch 'qatar/master'

* qatar/master:
  fate: Explicitly specify the rgb555le pixel format for the mss2 rgb555 tests

See: 34e90b2e6d66c511bea872bcb3a0e34b482b8c17
Merged-by: Michael Niedermayer <michaelni@gmx.at>

avformat/flvdec: initialize context before reading from it

Fixes use of uninitialized memory
Fixes: msan_uninit-mem_7f9b8387069e_5377_flv_with_pcm_s16be_audio_track.flv
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

nut: Support XYZ12 rawvideo

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

swscale: fix rgb48Toxyz12() endianness

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/mss2: fix chroma dimensions

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

fate: Explicitly specify the rgb555le pixel format for the mss2 rgb555 tests

This fixes these fate tests on big endian.

Signed-off-by: Martin Storsjö <martin@martin.st>

avformat/avisynth simplify packet allocation

Reviewed-by: Stephen Hutchinson <qyot27@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fix libopenjpeg colour range adjust for 8<bpp<16.

Fixes ticket #3284.

Reviewed-by: Michael Bradshaw

Support signed j2k images via libopenjpeg.

Fixes ticket #3283.

Reviewed-by: Michael Bradshaw

Unscaled 16bit packed RGB to planar GBR converter.

Fixes r210(rgb48) -> ffv1 (gbrp10) -> r210 roundtrip
as reported by forum user JasonCA.

fate: fix MSS2 tests on big endian

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Merge remote-tracking branch 'qatar/master'

* qatar/master:
  avutil: do not use avcodec header in frame.h

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit 'abb5e37f64c48bba8bd0fde2bada0f7544defa24'

* commit 'abb5e37f64c48bba8bd0fde2bada0f7544defa24':
  avfilter: fix leaks on error in ff_filter_frame

Conflicts:
libavfilter/avfilter.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '5655732c77f2df9c1dfbddc30cd4844390a6921f'

* commit '5655732c77f2df9c1dfbddc30cd4844390a6921f':
  avfilter: add needs_writable field to the internal AVFilterPad structure

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit 'd44bd7fb27d4121512eae0ffce5e48bac25e82dc'

* commit 'd44bd7fb27d4121512eae0ffce5e48bac25e82dc':
  avfilter: add documentation for needs_writable

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '11d704dc94c029058ac366f85951f6728638fb9c'

* commit '11d704dc94c029058ac366f85951f6728638fb9c':
  fate: add framepack filter test

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit 'a7d0e7ead95e584c4866617f046b2493066975c6'

* commit 'a7d0e7ead95e584c4866617f046b2493066975c6':
  lavfi: add framepack filter

Conflicts:
Changelog
doc/filters.texi
libavfilter/allfilters.c
libavfilter/version.h

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit 'aa69cbc9e08281db6ecb98a2c8b4f60e8b12e558'

* commit 'aa69cbc9e08281db6ecb98a2c8b4f60e8b12e558':
  flac muxer: add option to disable writing the global header

Conflicts:
libavformat/version.h

Merged-by: Michael Niedermayer <michaelni@gmx.at>

avformat/mxfdec: check avio_read(UID) result

Fixes use of uninitialized memory
Fixes: msan_uninit-mem_7fc9ba2fd98e_82_02785736.mxf
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/nutdec: fix packet end clearing

The code was buggy, using the wrong variable, also it missed the case
where the packet become smaller due to sidedata/metadata being extracted
which left a few bytes uninitialized

Fixes use of uninitialized memory
Fixed: msan_uninit-mem_7f6abbe44530_6838_mewmew_vorbis_ssa.nut
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/rmdec: when reading audio blocks, dont leave holes when reading fails

The fate test is changed because the reference file depends on the use of
non cleared data at the very
end. Alternatively we could upload a new reference file, though that would
then have to be changed every time the handling of a truncated frame changes
or theres a change to error concealment, each time adding a new file ...

Fixes use of uninitialized memory
Fixed: msan_uninit-mem_7f3c02b81363_2787_RLG2_19.rm
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avutil: do not use avcodec header in frame.h

avfilter: fix leaks on error in ff_filter_frame

avfilter: add needs_writable field to the internal AVFilterPad structure

avfilter: add documentation for needs_writable

fate: add framepack filter test

lavfi: add framepack filter

flac muxer: add option to disable writing the global header

The global header is not explicitly required for playback, and
omitting it allows for simple concatenation.

avcodec/mjpegdec: allow lowres with amv

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/mjpegdec: use the AVFrame width/height for flipping

This makes it possible to use the code with lowres

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/mjpegdec: remove new flipping code

It stored images wrong in the user provided buffers (that is you would
end up with a wrongly flipped image if you used direct rendering).
Also it used wrong dimensions as noticed by ubitux

Enable the old code unconditionally so flipping works correctly
again.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Merge remote-tracking branch 'qatar/master'

* qatar/master:
  ATRAC3+ decoder

Conflicts:
Changelog
configure
libavcodec/atrac3plus.c
libavcodec/atrac3plus.h
libavcodec/atrac3plus_data.h
libavcodec/atrac3plusdec.c
libavcodec/atrac3plusdsp.c
libavcodec/version.h

See: 2e1fb96af36348479e14aa85f445915ea09e70fa
Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '4c642d8d98703faf52983243098f35865e15b312'

* commit '4c642d8d98703faf52983243098f35865e15b312':
  x86: hpeldsp: Add missing av_cold attribute to init function

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit 'b0be1ae792ac8bbfb0fc7b9b9cb39eaf0feb489b'

* commit 'b0be1ae792ac8bbfb0fc7b9b9cb39eaf0feb489b':
  x86: avcodec: Add a bunch of missing #includes for av_cold

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '4b48201d4d9d57a2a1ecf8504bc928aff83cb99a'

* commit '4b48201d4d9d57a2a1ecf8504bc928aff83cb99a':
  ac3tab.h: #include the correct headers

Merged-by: Michael Niedermayer <michaelni@gmx.at>

ATRAC3+ decoder

Cleanup by Diego Biurrun.

Signed-off-by: Kostya Shishkov <kostya.shishkov@gmail.com>

doc/examples/muxing: Fixes frame initialization.

Fixes use of the example with encoders which use tha AVFrame w/h/pix_fmt fields
FFV1 is one of these codecs

We cannot easily workaround the not set fields in common code because the API
has AVFrame constant for the encoders.
Alternatives would be to fix the API or to duplicate the struct and fill in
missing fields. Or as is to require all user apps to set this correctly and
maybe simplify for that case

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Merge remote-tracking branch 'qatar/master'

* qatar/master:
  mjpegdec: apply flipping after decoding, not before

Conflicts:
libavcodec/mjpegdec.c
libavcodec/mjpegdec.h

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit 'e588615d938f8581f0d6f3771662d08cadfc00de'

* commit 'e588615d938f8581f0d6f3771662d08cadfc00de':
  hevc: fix decoding of one PU wide files

Conflicts:
libavcodec/hevc.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>

x86: hpeldsp: Add missing av_cold attribute to init function

x86: avcodec: Add a bunch of missing #includes for av_cold

ac3tab.h: #include the correct headers

Merge commit 'a246af86e47044dec243db2f708cc125d608ea78'

* commit 'a246af86e47044dec243db2f708cc125d608ea78':
  FATE: add MSS2 tests

Merged-by: Michael Niedermayer <michaelni@gmx.at>