lavu,lavfi,lavd: do not hardcode AV_PIX_FMT_NB value when setting pixel format max...

lavu,lavfi,lavd: do not hardcode AV_PIX_FMT_NB value when setting pixel format max value

The constant may change in libavutil but the library may be compiled
against an older version, thus rejecting a value which is otherwise
supported by the new libavutil.

INT_MAX is used here to denote the max allowed value for a pixel format.

The opt-test code is changed to provide a valid reference example.