Fix updating condition for the probe_size variable in the internal

Fix updating condition for the probe_size variable in the internal
loop of ff_probe_input_buffer(), making sure that probe_size is always
set to probe_max_size in the last iteration.

Also make the function return an error if we get to the max probe
length and still cannot figure out what the format is.

Patch by Micah Galizia micahgalizia A gmail D com.

Originally committed as revision 22688 to svn://svn.ffmpeg.org/ffmpeg/trunk