Actually return with an error condition if we're being asked to deal with too

Actually return with an error condition if we're being asked to deal with too
many reference frames. Also check max num ref frames against our internal
ref buffer sizes.
Part of fix for roundup issue 281

Originally committed as revision 11215 to svn://svn.ffmpeg.org/ffmpeg/trunk