ac3: validate end in ff_ac3_bit_alloc_calc_mask

This fixes an invalid read if end is 0:
     band_end   = ff_ac3_bin_to_band_tab[end-1] + 1;

Depending on what is before the array, this can cause stack smashing,
when band_end becomes too large.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit bc4fee7f2a51635fa3c0f61d1e5164da1efeded3)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

diff --git a/libavcodec/ac3.c b/libavcodec/ac3.c
index 29e132f..8d39bbe 100644 (file)
--- a/libavcodec/ac3.c
+++ b/libavcodec/ac3.c
@@ -131,6 +131,9 @@ int ff_ac3_bit_alloc_calc_mask(AC3BitAllocParameters *s, int16_t *band_psd,
     int band_start, band_end, begin, end1;
     int lowcomp, fastleak, slowleak;
 
+    if (end <= 0)
+        return AVERROR_INVALIDDATA;
+
     /* excitation function */
     band_start = ff_ac3_bin_to_band_tab[start];
     band_end   = ff_ac3_bin_to_band_tab[end-1] + 1;