2 * Copyright (C) 2012 The Android Open Source Project
4 * Licensed under the Apache License, Version 2.0 (the "License");
5 * you may not use this file except in compliance with the License.
6 * You may obtain a copy of the License at
8 * http://www.apache.org/licenses/LICENSE-2.0
10 * Unless required by applicable law or agreed to in writing, software
11 * distributed under the License is distributed on an "AS IS" BASIS,
12 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 * See the License for the specific language governing permissions and
14 * limitations under the License.
17 package android.media;
19 import android.annotation.IntDef;
20 import android.annotation.NonNull;
21 import android.annotation.Nullable;
22 import android.graphics.ImageFormat;
23 import android.graphics.Rect;
24 import android.graphics.SurfaceTexture;
25 import android.media.MediaCodecInfo.CodecCapabilities;
26 import android.os.Bundle;
27 import android.os.Handler;
28 import android.os.IBinder;
29 import android.os.IHwBinder;
30 import android.os.Looper;
31 import android.os.Message;
32 import android.os.PersistableBundle;
33 import android.view.Surface;
35 import java.io.IOException;
36 import java.lang.annotation.Retention;
37 import java.lang.annotation.RetentionPolicy;
38 import java.nio.ByteBuffer;
39 import java.nio.ByteOrder;
40 import java.nio.ReadOnlyBufferException;
41 import java.util.Arrays;
42 import java.util.HashMap;
46 MediaCodec class can be used to access low-level media codecs, i.e. encoder/decoder components.
47 It is part of the Android low-level multimedia support infrastructure (normally used together
48 with {@link MediaExtractor}, {@link MediaSync}, {@link MediaMuxer}, {@link MediaCrypto},
49 {@link MediaDrm}, {@link Image}, {@link Surface}, and {@link AudioTrack}.)
51 <center><object style="width: 540px; height: 205px;" type="image/svg+xml"
52 data="../../../images/media/mediacodec_buffers.svg"><img
53 src="../../../images/media/mediacodec_buffers.png" style="width: 540px; height: 205px"
54 alt="MediaCodec buffer flow diagram"></object></center>
56 In broad terms, a codec processes input data to generate output data. It processes data
57 asynchronously and uses a set of input and output buffers. At a simplistic level, you request
58 (or receive) an empty input buffer, fill it up with data and send it to the codec for
59 processing. The codec uses up the data and transforms it into one of its empty output buffers.
60 Finally, you request (or receive) a filled output buffer, consume its contents and release it
65 Codecs operate on three kinds of data: compressed data, raw audio data and raw video data.
66 All three kinds of data can be processed using {@link ByteBuffer ByteBuffers}, but you should use
67 a {@link Surface} for raw video data to improve codec performance. Surface uses native video
68 buffers without mapping or copying them to ByteBuffers; thus, it is much more efficient.
69 You normally cannot access the raw video data when using a Surface, but you can use the
70 {@link ImageReader} class to access unsecured decoded (raw) video frames. This may still be more
71 efficient than using ByteBuffers, as some native buffers may be mapped into {@linkplain
72 ByteBuffer#isDirect direct} ByteBuffers. When using ByteBuffer mode, you can access raw video
73 frames using the {@link Image} class and {@link #getInputImage getInput}/{@link #getOutputImage
76 <h4>Compressed Buffers</h4>
78 Input buffers (for decoders) and output buffers (for encoders) contain compressed data according
79 to the {@linkplain MediaFormat#KEY_MIME format's type}. For video types this is normally a single
80 compressed video frame. For audio data this is normally a single access unit (an encoded audio
81 segment typically containing a few milliseconds of audio as dictated by the format type), but
82 this requirement is slightly relaxed in that a buffer may contain multiple encoded access units
83 of audio. In either case, buffers do not start or end on arbitrary byte boundaries, but rather on
84 frame/access unit boundaries unless they are flagged with {@link #BUFFER_FLAG_PARTIAL_FRAME}.
86 <h4>Raw Audio Buffers</h4>
88 Raw audio buffers contain entire frames of PCM audio data, which is one sample for each channel
89 in channel order. Each sample is a {@linkplain AudioFormat#ENCODING_PCM_16BIT 16-bit signed
90 integer in native byte order}.
92 <pre class=prettyprint>
93 short[] getSamplesForChannel(MediaCodec codec, int bufferId, int channelIx) {
94 ByteBuffer outputBuffer = codec.getOutputBuffer(bufferId);
95 MediaFormat format = codec.getOutputFormat(bufferId);
96 ShortBuffer samples = outputBuffer.order(ByteOrder.nativeOrder()).asShortBuffer();
97 int numChannels = formet.getInteger(MediaFormat.KEY_CHANNEL_COUNT);
98 if (channelIx < 0 || channelIx >= numChannels) {
101 short[] res = new short[samples.remaining() / numChannels];
102 for (int i = 0; i < res.length; ++i) {
103 res[i] = samples.get(i * numChannels + channelIx);
108 <h4>Raw Video Buffers</h4>
110 In ByteBuffer mode video buffers are laid out according to their {@linkplain
111 MediaFormat#KEY_COLOR_FORMAT color format}. You can get the supported color formats as an array
112 from {@link #getCodecInfo}{@code .}{@link MediaCodecInfo#getCapabilitiesForType
113 getCapabilitiesForType(…)}{@code .}{@link CodecCapabilities#colorFormats colorFormats}.
114 Video codecs may support three kinds of color formats:
116 <li><strong>native raw video format:</strong> This is marked by {@link
117 CodecCapabilities#COLOR_FormatSurface} and it can be used with an input or output Surface.</li>
118 <li><strong>flexible YUV buffers</strong> (such as {@link
119 CodecCapabilities#COLOR_FormatYUV420Flexible}): These can be used with an input/output Surface,
120 as well as in ByteBuffer mode, by using {@link #getInputImage getInput}/{@link #getOutputImage
121 OutputImage(int)}.</li>
122 <li><strong>other, specific formats:</strong> These are normally only supported in ByteBuffer
123 mode. Some color formats are vendor specific. Others are defined in {@link CodecCapabilities}.
124 For color formats that are equivalent to a flexible format, you can still use {@link
125 #getInputImage getInput}/{@link #getOutputImage OutputImage(int)}.</li>
128 All video codecs support flexible YUV 4:2:0 buffers since {@link
129 android.os.Build.VERSION_CODES#LOLLIPOP_MR1}.
131 <h4>Accessing Raw Video ByteBuffers on Older Devices</h4>
133 Prior to {@link android.os.Build.VERSION_CODES#LOLLIPOP} and {@link Image} support, you need to
134 use the {@link MediaFormat#KEY_STRIDE} and {@link MediaFormat#KEY_SLICE_HEIGHT} output format
135 values to understand the layout of the raw output buffers.
137 Note that on some devices the slice-height is advertised as 0. This could mean either that the
138 slice-height is the same as the frame height, or that the slice-height is the frame height
139 aligned to some value (usually a power of 2). Unfortunately, there is no standard and simple way
140 to tell the actual slice height in this case. Furthermore, the vertical stride of the {@code U}
141 plane in planar formats is also not specified or defined, though usually it is half of the slice
144 The {@link MediaFormat#KEY_WIDTH} and {@link MediaFormat#KEY_HEIGHT} keys specify the size of the
145 video frames; however, for most encondings the video (picture) only occupies a portion of the
146 video frame. This is represented by the 'crop rectangle'.
148 You need to use the following keys to get the crop rectangle of raw output images from the
149 {@linkplain #getOutputFormat output format}. If these keys are not present, the video occupies the
150 entire video frame.The crop rectangle is understood in the context of the output frame
151 <em>before</em> applying any {@linkplain MediaFormat#KEY_ROTATION rotation}.
152 <table style="width: 0%">
162 <td>{@code "crop-left"}</td>
164 <td>The left-coordinate (x) of the crop rectangle</td>
166 <td>{@code "crop-top"}</td>
168 <td>The top-coordinate (y) of the crop rectangle</td>
170 <td>{@code "crop-right"}</td>
172 <td>The right-coordinate (x) <strong>MINUS 1</strong> of the crop rectangle</td>
174 <td>{@code "crop-bottom"}</td>
176 <td>The bottom-coordinate (y) <strong>MINUS 1</strong> of the crop rectangle</td>
179 The right and bottom coordinates can be understood as the coordinates of the right-most
180 valid column/bottom-most valid row of the cropped output image.
186 The size of the video frame (before rotation) can be calculated as such:
187 <pre class=prettyprint>
188 MediaFormat format = decoder.getOutputFormat(…);
189 int width = format.getInteger(MediaFormat.KEY_WIDTH);
190 if (format.containsKey("crop-left") && format.containsKey("crop-right")) {
191 width = format.getInteger("crop-right") + 1 - format.getInteger("crop-left");
193 int height = format.getInteger(MediaFormat.KEY_HEIGHT);
194 if (format.containsKey("crop-top") && format.containsKey("crop-bottom")) {
195 height = format.getInteger("crop-bottom") + 1 - format.getInteger("crop-top");
199 Also note that the meaning of {@link BufferInfo#offset BufferInfo.offset} was not consistent across
200 devices. On some devices the offset pointed to the top-left pixel of the crop rectangle, while on
201 most devices it pointed to the top-left pixel of the entire frame.
205 During its life a codec conceptually exists in one of three states: Stopped, Executing or
206 Released. The Stopped collective state is actually the conglomeration of three states:
207 Uninitialized, Configured and Error, whereas the Executing state conceptually progresses through
208 three sub-states: Flushed, Running and End-of-Stream.
210 <center><object style="width: 516px; height: 353px;" type="image/svg+xml"
211 data="../../../images/media/mediacodec_states.svg"><img
212 src="../../../images/media/mediacodec_states.png" style="width: 519px; height: 356px"
213 alt="MediaCodec state diagram"></object></center>
215 When you create a codec using one of the factory methods, the codec is in the Uninitialized
216 state. First, you need to configure it via {@link #configure configure(…)}, which brings
217 it to the Configured state, then call {@link #start} to move it to the Executing state. In this
218 state you can process data through the buffer queue manipulation described above.
220 The Executing state has three sub-states: Flushed, Running and End-of-Stream. Immediately after
221 {@link #start} the codec is in the Flushed sub-state, where it holds all the buffers. As soon
222 as the first input buffer is dequeued, the codec moves to the Running sub-state, where it spends
223 most of its life. When you queue an input buffer with the {@linkplain #BUFFER_FLAG_END_OF_STREAM
224 end-of-stream marker}, the codec transitions to the End-of-Stream sub-state. In this state the
225 codec no longer accepts further input buffers, but still generates output buffers until the
226 end-of-stream is reached on the output. You can move back to the Flushed sub-state at any time
227 while in the Executing state using {@link #flush}.
229 Call {@link #stop} to return the codec to the Uninitialized state, whereupon it may be configured
230 again. When you are done using a codec, you must release it by calling {@link #release}.
232 On rare occasions the codec may encounter an error and move to the Error state. This is
233 communicated using an invalid return value from a queuing operation, or sometimes via an
234 exception. Call {@link #reset} to make the codec usable again. You can call it from any state to
235 move the codec back to the Uninitialized state. Otherwise, call {@link #release} to move to the
236 terminal Released state.
240 Use {@link MediaCodecList} to create a MediaCodec for a specific {@link MediaFormat}. When
241 decoding a file or a stream, you can get the desired format from {@link
242 MediaExtractor#getTrackFormat MediaExtractor.getTrackFormat}. Inject any specific features that
243 you want to add using {@link MediaFormat#setFeatureEnabled MediaFormat.setFeatureEnabled}, then
244 call {@link MediaCodecList#findDecoderForFormat MediaCodecList.findDecoderForFormat} to get the
245 name of a codec that can handle that specific media format. Finally, create the codec using
246 {@link #createByCodecName}.
248 <strong>Note:</strong> On {@link android.os.Build.VERSION_CODES#LOLLIPOP}, the format to
249 {@code MediaCodecList.findDecoder}/{@code EncoderForFormat} must not contain a {@linkplain
250 MediaFormat#KEY_FRAME_RATE frame rate}. Use
251 <code class=prettyprint>format.setString(MediaFormat.KEY_FRAME_RATE, null)</code>
252 to clear any existing frame rate setting in the format.
254 You can also create the preferred codec for a specific MIME type using {@link
255 #createDecoderByType createDecoder}/{@link #createEncoderByType EncoderByType(String)}.
256 This, however, cannot be used to inject features, and may create a codec that cannot handle the
257 specific desired media format.
259 <h4>Creating secure decoders</h4>
261 On versions {@link android.os.Build.VERSION_CODES#KITKAT_WATCH} and earlier, secure codecs might
262 not be listed in {@link MediaCodecList}, but may still be available on the system. Secure codecs
263 that exist can be instantiated by name only, by appending {@code ".secure"} to the name of a
264 regular codec (the name of all secure codecs must end in {@code ".secure"}.) {@link
265 #createByCodecName} will throw an {@code IOException} if the codec is not present on the system.
267 From {@link android.os.Build.VERSION_CODES#LOLLIPOP} onwards, you should use the {@link
268 CodecCapabilities#FEATURE_SecurePlayback} feature in the media format to create a secure decoder.
270 <h3>Initialization</h3>
272 After creating the codec, you can set a callback using {@link #setCallback setCallback} if you
273 want to process data asynchronously. Then, {@linkplain #configure configure} the codec using the
274 specific media format. This is when you can specify the output {@link Surface} for video
275 producers – codecs that generate raw video data (e.g. video decoders). This is also when
276 you can set the decryption parameters for secure codecs (see {@link MediaCrypto}). Finally, since
277 some codecs can operate in multiple modes, you must specify whether you want it to work as a
278 decoder or an encoder.
280 Since {@link android.os.Build.VERSION_CODES#LOLLIPOP}, you can query the resulting input and
281 output format in the Configured state. You can use this to verify the resulting configuration,
282 e.g. color formats, before starting the codec.
284 If you want to process raw input video buffers natively with a video consumer – a codec
285 that processes raw video input, such as a video encoder – create a destination Surface for
286 your input data using {@link #createInputSurface} after configuration. Alternately, set up the
287 codec to use a previously created {@linkplain #createPersistentInputSurface persistent input
288 surface} by calling {@link #setInputSurface}.
290 <h4 id=CSD><a name="CSD"></a>Codec-specific Data</h4>
292 Some formats, notably AAC audio and MPEG4, H.264 and H.265 video formats require the actual data
293 to be prefixed by a number of buffers containing setup data, or codec specific data. When
294 processing such compressed formats, this data must be submitted to the codec after {@link
295 #start} and before any frame data. Such data must be marked using the flag {@link
296 #BUFFER_FLAG_CODEC_CONFIG} in a call to {@link #queueInputBuffer queueInputBuffer}.
298 Codec-specific data can also be included in the format passed to {@link #configure configure} in
299 ByteBuffer entries with keys "csd-0", "csd-1", etc. These keys are always included in the track
300 {@link MediaFormat} obtained from the {@link MediaExtractor#getTrackFormat MediaExtractor}.
301 Codec-specific data in the format is automatically submitted to the codec upon {@link #start};
302 you <strong>MUST NOT</strong> submit this data explicitly. If the format did not contain codec
303 specific data, you can choose to submit it using the specified number of buffers in the correct
304 order, according to the format requirements. In case of H.264 AVC, you can also concatenate all
305 codec-specific data and submit it as a single codec-config buffer.
307 Android uses the following codec-specific data buffers. These are also required to be set in
308 the track format for proper {@link MediaMuxer} track configuration. Each parameter set and the
309 codec-specific-data sections marked with (<sup>*</sup>) must start with a start code of
310 {@code "\x00\x00\x00\x01"}.
312 <style>td.NA { background: #ccc; } .mid > tr > td { vertical-align: middle; }</style>
316 <th>CSD buffer #0</th>
317 <th>CSD buffer #1</th>
318 <th>CSD buffer #2</th>
323 <td>Decoder-specific information from ESDS<sup>*</sup></td>
324 <td class=NA>Not Used</td>
325 <td class=NA>Not Used</td>
329 <td>Identification header</td>
330 <td>Setup header</td>
331 <td class=NA>Not Used</td>
335 <td>Identification header</td>
336 <td>Pre-skip in nanosecs<br>
337 (unsigned 64-bit {@linkplain ByteOrder#nativeOrder native-order} integer.)<br>
338 This overrides the pre-skip value in the identification header.</td>
339 <td>Seek Pre-roll in nanosecs<br>
340 (unsigned 64-bit {@linkplain ByteOrder#nativeOrder native-order} integer.)</td>
344 <td>"fLaC", the FLAC stream marker in ASCII,<br>
345 followed by the STREAMINFO block (the mandatory metadata block),<br>
346 optionally followed by any number of other metadata blocks</td>
347 <td class=NA>Not Used</td>
348 <td class=NA>Not Used</td>
352 <td>Decoder-specific information from ESDS<sup>*</sup></td>
353 <td class=NA>Not Used</td>
354 <td class=NA>Not Used</td>
358 <td>SPS (Sequence Parameter Sets<sup>*</sup>)</td>
359 <td>PPS (Picture Parameter Sets<sup>*</sup>)</td>
360 <td class=NA>Not Used</td>
364 <td>VPS (Video Parameter Sets<sup>*</sup>) +<br>
365 SPS (Sequence Parameter Sets<sup>*</sup>) +<br>
366 PPS (Picture Parameter Sets<sup>*</sup>)</td>
367 <td class=NA>Not Used</td>
368 <td class=NA>Not Used</td>
372 <td>VP9 <a href="http://wiki.webmproject.org/vp9-codecprivate">CodecPrivate</a> Data
374 <td class=NA>Not Used</td>
375 <td class=NA>Not Used</td>
381 <strong>Note:</strong> care must be taken if the codec is flushed immediately or shortly
382 after start, before any output buffer or output format change has been returned, as the codec
383 specific data may be lost during the flush. You must resubmit the data using buffers marked with
384 {@link #BUFFER_FLAG_CODEC_CONFIG} after such flush to ensure proper codec operation.
386 Encoders (or codecs that generate compressed data) will create and return the codec specific data
387 before any valid output buffer in output buffers marked with the {@linkplain
388 #BUFFER_FLAG_CODEC_CONFIG codec-config flag}. Buffers containing codec-specific-data have no
389 meaningful timestamps.
391 <h3>Data Processing</h3>
393 Each codec maintains a set of input and output buffers that are referred to by a buffer-ID in
394 API calls. After a successful call to {@link #start} the client "owns" neither input nor output
395 buffers. In synchronous mode, call {@link #dequeueInputBuffer dequeueInput}/{@link
396 #dequeueOutputBuffer OutputBuffer(…)} to obtain (get ownership of) an input or output
397 buffer from the codec. In asynchronous mode, you will automatically receive available buffers via
398 the {@link Callback#onInputBufferAvailable MediaCodec.Callback.onInput}/{@link
399 Callback#onOutputBufferAvailable OutputBufferAvailable(…)} callbacks.
401 Upon obtaining an input buffer, fill it with data and submit it to the codec using {@link
402 #queueInputBuffer queueInputBuffer} – or {@link #queueSecureInputBuffer
403 queueSecureInputBuffer} if using decryption. Do not submit multiple input buffers with the same
404 timestamp (unless it is <a href="#CSD">codec-specific data</a> marked as such).
406 The codec in turn will return a read-only output buffer via the {@link
407 Callback#onOutputBufferAvailable onOutputBufferAvailable} callback in asynchronous mode, or in
408 response to a {@link #dequeueOutputBuffer dequeueOutputBuffer} call in synchronous mode. After the
409 output buffer has been processed, call one of the {@link #releaseOutputBuffer
410 releaseOutputBuffer} methods to return the buffer to the codec.
412 While you are not required to resubmit/release buffers immediately to the codec, holding onto
413 input and/or output buffers may stall the codec, and this behavior is device dependent.
414 <strong>Specifically, it is possible that a codec may hold off on generating output buffers until
415 <em>all</em> outstanding buffers have been released/resubmitted.</strong> Therefore, try to
416 hold onto to available buffers as little as possible.
418 Depending on the API version, you can process data in three ways:
422 <th>Processing Mode</th>
423 <th>API version <= 20<br>Jelly Bean/KitKat</th>
424 <th>API version >= 21<br>Lollipop and later</th>
429 <td>Synchronous API using buffer arrays</td>
434 <td>Synchronous API using buffers</td>
435 <td class=NA>Not Available</td>
439 <td>Asynchronous API using buffers</td>
440 <td class=NA>Not Available</td>
446 <h4>Asynchronous Processing using Buffers</h4>
448 Since {@link android.os.Build.VERSION_CODES#LOLLIPOP}, the preferred method is to process data
449 asynchronously by setting a callback before calling {@link #configure configure}. Asynchronous
450 mode changes the state transitions slightly, because you must call {@link #start} after {@link
451 #flush} to transition the codec to the Running sub-state and start receiving input buffers.
452 Similarly, upon an initial call to {@code start} the codec will move directly to the Running
453 sub-state and start passing available input buffers via the callback.
455 <center><object style="width: 516px; height: 353px;" type="image/svg+xml"
456 data="../../../images/media/mediacodec_async_states.svg"><img
457 src="../../../images/media/mediacodec_async_states.png" style="width: 516px; height: 353px"
458 alt="MediaCodec state diagram for asynchronous operation"></object></center>
460 MediaCodec is typically used like this in asynchronous mode:
461 <pre class=prettyprint>
462 MediaCodec codec = MediaCodec.createByCodecName(name);
463 MediaFormat mOutputFormat; // member variable
464 codec.setCallback(new MediaCodec.Callback() {
466 void onInputBufferAvailable(MediaCodec mc, int inputBufferId) {
467 ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferId);
468 // fill inputBuffer with valid data
470 codec.queueInputBuffer(inputBufferId, …);
474 void onOutputBufferAvailable(MediaCodec mc, int outputBufferId, …) {
475 ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
476 MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
477 // bufferFormat is equivalent to mOutputFormat
478 // outputBuffer is ready to be processed or rendered.
480 codec.releaseOutputBuffer(outputBufferId, …);
484 void onOutputFormatChanged(MediaCodec mc, MediaFormat format) {
485 // Subsequent data will conform to new format.
486 // Can ignore if using getOutputFormat(outputBufferId)
487 mOutputFormat = format; // option B
491 void onError(…) {
495 codec.configure(format, …);
496 mOutputFormat = codec.getOutputFormat(); // option B
498 // wait for processing to complete
500 codec.release();</pre>
502 <h4>Synchronous Processing using Buffers</h4>
504 Since {@link android.os.Build.VERSION_CODES#LOLLIPOP}, you should retrieve input and output
505 buffers using {@link #getInputBuffer getInput}/{@link #getOutputBuffer OutputBuffer(int)} and/or
506 {@link #getInputImage getInput}/{@link #getOutputImage OutputImage(int)} even when using the
507 codec in synchronous mode. This allows certain optimizations by the framework, e.g. when
508 processing dynamic content. This optimization is disabled if you call {@link #getInputBuffers
509 getInput}/{@link #getOutputBuffers OutputBuffers()}.
512 <strong>Note:</strong> do not mix the methods of using buffers and buffer arrays at the same
513 time. Specifically, only call {@code getInput}/{@code OutputBuffers} directly after {@link
514 #start} or after having dequeued an output buffer ID with the value of {@link
515 #INFO_OUTPUT_FORMAT_CHANGED}.
517 MediaCodec is typically used like this in synchronous mode:
519 MediaCodec codec = MediaCodec.createByCodecName(name);
520 codec.configure(format, …);
521 MediaFormat outputFormat = codec.getOutputFormat(); // option B
524 int inputBufferId = codec.dequeueInputBuffer(timeoutUs);
525 if (inputBufferId >= 0) {
526 ByteBuffer inputBuffer = codec.getInputBuffer(…);
527 // fill inputBuffer with valid data
529 codec.queueInputBuffer(inputBufferId, …);
531 int outputBufferId = codec.dequeueOutputBuffer(…);
532 if (outputBufferId >= 0) {
533 ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
534 MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
535 // bufferFormat is identical to outputFormat
536 // outputBuffer is ready to be processed or rendered.
538 codec.releaseOutputBuffer(outputBufferId, …);
539 } else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
540 // Subsequent data will conform to new format.
541 // Can ignore if using getOutputFormat(outputBufferId)
542 outputFormat = codec.getOutputFormat(); // option B
546 codec.release();</pre>
548 <h4>Synchronous Processing using Buffer Arrays (deprecated)</h4>
550 In versions {@link android.os.Build.VERSION_CODES#KITKAT_WATCH} and before, the set of input and
551 output buffers are represented by the {@code ByteBuffer[]} arrays. After a successful call to
552 {@link #start}, retrieve the buffer arrays using {@link #getInputBuffers getInput}/{@link
553 #getOutputBuffers OutputBuffers()}. Use the buffer ID-s as indices into these arrays (when
554 non-negative), as demonstrated in the sample below. Note that there is no inherent correlation
555 between the size of the arrays and the number of input and output buffers used by the system,
556 although the array size provides an upper bound.
558 MediaCodec codec = MediaCodec.createByCodecName(name);
559 codec.configure(format, …);
561 ByteBuffer[] inputBuffers = codec.getInputBuffers();
562 ByteBuffer[] outputBuffers = codec.getOutputBuffers();
564 int inputBufferId = codec.dequeueInputBuffer(…);
565 if (inputBufferId >= 0) {
566 // fill inputBuffers[inputBufferId] with valid data
568 codec.queueInputBuffer(inputBufferId, …);
570 int outputBufferId = codec.dequeueOutputBuffer(…);
571 if (outputBufferId >= 0) {
572 // outputBuffers[outputBufferId] is ready to be processed or rendered.
574 codec.releaseOutputBuffer(outputBufferId, …);
575 } else if (outputBufferId == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
576 outputBuffers = codec.getOutputBuffers();
577 } else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
578 // Subsequent data will conform to new format.
579 MediaFormat format = codec.getOutputFormat();
583 codec.release();</pre>
585 <h4>End-of-stream Handling</h4>
587 When you reach the end of the input data, you must signal it to the codec by specifying the
588 {@link #BUFFER_FLAG_END_OF_STREAM} flag in the call to {@link #queueInputBuffer
589 queueInputBuffer}. You can do this on the last valid input buffer, or by submitting an additional
590 empty input buffer with the end-of-stream flag set. If using an empty buffer, the timestamp will
593 The codec will continue to return output buffers until it eventually signals the end of the
594 output stream by specifying the same end-of-stream flag in the {@link BufferInfo} set in {@link
595 #dequeueOutputBuffer dequeueOutputBuffer} or returned via {@link Callback#onOutputBufferAvailable
596 onOutputBufferAvailable}. This can be set on the last valid output buffer, or on an empty buffer
597 after the last valid output buffer. The timestamp of such empty buffer should be ignored.
599 Do not submit additional input buffers after signaling the end of the input stream, unless the
600 codec has been flushed, or stopped and restarted.
602 <h4>Using an Output Surface</h4>
604 The data processing is nearly identical to the ByteBuffer mode when using an output {@link
605 Surface}; however, the output buffers will not be accessible, and are represented as {@code null}
606 values. E.g. {@link #getOutputBuffer getOutputBuffer}/{@link #getOutputImage Image(int)} will
607 return {@code null} and {@link #getOutputBuffers} will return an array containing only {@code
610 When using an output Surface, you can select whether or not to render each output buffer on the
611 surface. You have three choices:
613 <li><strong>Do not render the buffer:</strong> Call {@link #releaseOutputBuffer(int, boolean)
614 releaseOutputBuffer(bufferId, false)}.</li>
615 <li><strong>Render the buffer with the default timestamp:</strong> Call {@link
616 #releaseOutputBuffer(int, boolean) releaseOutputBuffer(bufferId, true)}.</li>
617 <li><strong>Render the buffer with a specific timestamp:</strong> Call {@link
618 #releaseOutputBuffer(int, long) releaseOutputBuffer(bufferId, timestamp)}.</li>
621 Since {@link android.os.Build.VERSION_CODES#M}, the default timestamp is the {@linkplain
622 BufferInfo#presentationTimeUs presentation timestamp} of the buffer (converted to nanoseconds).
623 It was not defined prior to that.
625 Also since {@link android.os.Build.VERSION_CODES#M}, you can change the output Surface
626 dynamically using {@link #setOutputSurface setOutputSurface}.
628 <h4>Transformations When Rendering onto Surface</h4>
630 If the codec is configured into Surface mode, any crop rectangle, {@linkplain
631 MediaFormat#KEY_ROTATION rotation} and {@linkplain #setVideoScalingMode video scaling
632 mode} will be automatically applied with one exception:
634 Prior to the {@link android.os.Build.VERSION_CODES#M} release, software decoders may not
635 have applied the rotation when being rendered onto a Surface. Unfortunately, there is no standard
636 and simple way to identify software decoders, or if they apply the rotation other than by trying
639 There are also some caveats.
641 Note that the pixel aspect ratio is not considered when displaying the output onto the
642 Surface. This means that if you are using {@link #VIDEO_SCALING_MODE_SCALE_TO_FIT} mode, you
643 must position the output Surface so that it has the proper final display aspect ratio. Conversely,
644 you can only use {@link #VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING} mode for content with
645 square pixels (pixel aspect ratio or 1:1).
647 Note also that as of {@link android.os.Build.VERSION_CODES#N} release, {@link
648 #VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING} mode may not work correctly for videos rotated
649 by 90 or 270 degrees.
651 When setting the video scaling mode, note that it must be reset after each time the output
652 buffers change. Since the {@link #INFO_OUTPUT_BUFFERS_CHANGED} event is deprecated, you can
653 do this after each time the output format changes.
655 <h4>Using an Input Surface</h4>
657 When using an input Surface, there are no accessible input buffers, as buffers are automatically
658 passed from the input surface to the codec. Calling {@link #dequeueInputBuffer
659 dequeueInputBuffer} will throw an {@code IllegalStateException}, and {@link #getInputBuffers}
660 returns a bogus {@code ByteBuffer[]} array that <strong>MUST NOT</strong> be written into.
662 Call {@link #signalEndOfInputStream} to signal end-of-stream. The input surface will stop
663 submitting data to the codec immediately after this call.
666 <h3>Seeking & Adaptive Playback Support</h3>
668 Video decoders (and in general codecs that consume compressed video data) behave differently
669 regarding seek and format change whether or not they support and are configured for adaptive
670 playback. You can check if a decoder supports {@linkplain
671 CodecCapabilities#FEATURE_AdaptivePlayback adaptive playback} via {@link
672 CodecCapabilities#isFeatureSupported CodecCapabilities.isFeatureSupported(String)}. Adaptive
673 playback support for video decoders is only activated if you configure the codec to decode onto a
676 <h4 id=KeyFrames><a name="KeyFrames"></a>Stream Boundary and Key Frames</h4>
678 It is important that the input data after {@link #start} or {@link #flush} starts at a suitable
679 stream boundary: the first frame must a key frame. A <em>key frame</em> can be decoded
680 completely on its own (for most codecs this means an I-frame), and no frames that are to be
681 displayed after a key frame refer to frames before the key frame.
683 The following table summarizes suitable key frames for various video formats.
688 <th>Suitable key frame</th>
694 <td>a suitable intraframe where no subsequent frames refer to frames prior to this frame.<br>
695 <i>(There is no specific name for such key frame.)</i></td>
706 <td>MPEG-4<br>H.263<br>MPEG-2</td>
707 <td>a suitable I-frame where no subsequent frames refer to frames prior to this frame.<br>
708 <i>(There is no specific name for such key frame.)</td>
713 <h4>For decoders that do not support adaptive playback (including when not decoding onto a
716 In order to start decoding data that is not adjacent to previously submitted data (i.e. after a
717 seek) you <strong>MUST</strong> flush the decoder. Since all output buffers are immediately
718 revoked at the point of the flush, you may want to first signal then wait for the end-of-stream
719 before you call {@code flush}. It is important that the input data after a flush starts at a
720 suitable stream boundary/key frame.
722 <strong>Note:</strong> the format of the data submitted after a flush must not change; {@link
723 #flush} does not support format discontinuities; for that, a full {@link #stop} - {@link
724 #configure configure(…)} - {@link #start} cycle is necessary.
727 <strong>Also note:</strong> if you flush the codec too soon after {@link #start} –
728 generally, before the first output buffer or output format change is received – you
729 will need to resubmit the codec-specific-data to the codec. See the <a
730 href="#CSD">codec-specific-data section</a> for more info.
732 <h4>For decoders that support and are configured for adaptive playback</h4>
734 In order to start decoding data that is not adjacent to previously submitted data (i.e. after a
735 seek) it is <em>not necessary</em> to flush the decoder; however, input data after the
736 discontinuity must start at a suitable stream boundary/key frame.
738 For some video formats - namely H.264, H.265, VP8 and VP9 - it is also possible to change the
739 picture size or configuration mid-stream. To do this you must package the entire new
740 codec-specific configuration data together with the key frame into a single buffer (including
741 any start codes), and submit it as a <strong>regular</strong> input buffer.
743 You will receive an {@link #INFO_OUTPUT_FORMAT_CHANGED} return value from {@link
744 #dequeueOutputBuffer dequeueOutputBuffer} or a {@link Callback#onOutputBufferAvailable
745 onOutputFormatChanged} callback just after the picture-size change takes place and before any
746 frames with the new size have been returned.
748 <strong>Note:</strong> just as the case for codec-specific data, be careful when calling
749 {@link #flush} shortly after you have changed the picture size. If you have not received
750 confirmation of the picture size change, you will need to repeat the request for the new picture
753 <h3>Error handling</h3>
755 The factory methods {@link #createByCodecName createByCodecName} and {@link #createDecoderByType
756 createDecoder}/{@link #createEncoderByType EncoderByType} throw {@code IOException} on failure
757 which you must catch or declare to pass up. MediaCodec methods throw {@code
758 IllegalStateException} when the method is called from a codec state that does not allow it; this
759 is typically due to incorrect application API usage. Methods involving secure buffers may throw
760 {@link CryptoException}, which has further error information obtainable from {@link
761 CryptoException#getErrorCode}.
763 Internal codec errors result in a {@link CodecException}, which may be due to media content
764 corruption, hardware failure, resource exhaustion, and so forth, even when the application is
765 correctly using the API. The recommended action when receiving a {@code CodecException}
766 can be determined by calling {@link CodecException#isRecoverable} and {@link
767 CodecException#isTransient}:
769 <li><strong>recoverable errors:</strong> If {@code isRecoverable()} returns true, then call
770 {@link #stop}, {@link #configure configure(…)}, and {@link #start} to recover.</li>
771 <li><strong>transient errors:</strong> If {@code isTransient()} returns true, then resources are
772 temporarily unavailable and the method may be retried at a later time.</li>
773 <li><strong>fatal errors:</strong> If both {@code isRecoverable()} and {@code isTransient()}
774 return false, then the {@code CodecException} is fatal and the codec must be {@linkplain #reset
775 reset} or {@linkplain #release released}.</li>
778 Both {@code isRecoverable()} and {@code isTransient()} do not return true at the same time.
780 <h2 id=History><a name="History"></a>Valid API Calls and API History</h2>
782 This sections summarizes the valid API calls in each state and the API history of the MediaCodec
783 class. For API version numbers, see {@link android.os.Build.VERSION_CODES}.
786 .api > tr > th, .api > tr > td { text-align: center; padding: 4px 4px; }
787 .api > tr > th { vertical-align: bottom; }
788 .api > tr > td { vertical-align: middle; }
789 .sml > tr > th, .sml > tr > td { text-align: center; padding: 2px 4px; }
790 .fn { text-align: left; }
791 .fn > code > a { font: 14px/19px Roboto Condensed, sans-serif; }
793 white-space: nowrap; background: none; border: none; vertical-align: bottom;
794 width: 30px; height: 83px;
797 transform: skew(-45deg, 0deg) translate(1px, -67px);
798 transform-origin: bottom left 0;
799 width: 30px; height: 20px;
801 .deg45 > div > div { border: 1px solid #ddd; background: #999; height: 90px; width: 42px; }
802 .deg45 > div > div > div { transform: skew(45deg, 0deg) translate(-55px, 55px) rotate(-45deg); }
805 <table align="right" style="width: 0%">
807 <tr><th>Symbol</th><th>Meaning</th></tr>
810 <tr><td>●</td><td>Supported</td></tr>
811 <tr><td>⁕</td><td>Semantics changed</td></tr>
812 <tr><td>○</td><td>Experimental support</td></tr>
813 <tr><td>[ ]</td><td>Deprecated</td></tr>
814 <tr><td>⎋</td><td>Restricted to surface input mode</td></tr>
815 <tr><td>⎆</td><td>Restricted to surface output mode</td></tr>
816 <tr><td>▧</td><td>Restricted to ByteBuffer input mode</td></tr>
817 <tr><td>↩</td><td>Restricted to synchronous mode</td></tr>
818 <tr><td>⇄</td><td>Restricted to asynchronous mode</td></tr>
819 <tr><td>( )</td><td>Can be called, but shouldn't</td></tr>
823 <table style="width: 100%;">
826 <th class=deg45><div><div style="background:#4285f4"><div>Uninitialized</div></div></div></th>
827 <th class=deg45><div><div style="background:#f4b400"><div>Configured</div></div></div></th>
828 <th class=deg45><div><div style="background:#e67c73"><div>Flushed</div></div></div></th>
829 <th class=deg45><div><div style="background:#0f9d58"><div>Running</div></div></div></th>
830 <th class=deg45><div><div style="background:#f7cb4d"><div>End of Stream</div></div></div></th>
831 <th class=deg45><div><div style="background:#db4437"><div>Error</div></div></div></th>
832 <th class=deg45><div><div style="background:#666"><div>Released</div></div></div></th>
834 <th colspan="8">SDK Version</th>
837 <th colspan="7">State</th>
858 <td class=fn>{@link #createByCodecName createByCodecName}</td>
876 <td class=fn>{@link #createDecoderByType createDecoderByType}</td>
894 <td class=fn>{@link #createEncoderByType createEncoderByType}</td>
912 <td class=fn>{@link #createPersistentInputSurface createPersistentInputSurface}</td>
930 <td class=fn>{@link #configure configure}</td>
948 <td class=fn>{@link #createInputSurface createInputSurface}</td>
966 <td class=fn>{@link #dequeueInputBuffer dequeueInputBuffer}</td>
972 <td>⁕▧↩</td>
973 <td>▧↩</td>
974 <td>▧↩</td>
984 <td class=fn>{@link #dequeueOutputBuffer dequeueOutputBuffer}</td>
990 <td>⁕↩</td>
1002 <td class=fn>{@link #flush flush}</td>
1020 <td class=fn>{@link #getCodecInfo getCodecInfo}</td>
1038 <td class=fn>{@link #getInputBuffer getInputBuffer}</td>
1056 <td class=fn>{@link #getInputBuffers getInputBuffers}</td>
1062 <td>[⁕↩]</td>
1074 <td class=fn>{@link #getInputFormat getInputFormat}</td>
1092 <td class=fn>{@link #getInputImage getInputImage}</td>
1110 <td class=fn>{@link #getName getName}</td>
1128 <td class=fn>{@link #getOutputBuffer getOutputBuffer}</td>
1146 <td class=fn>{@link #getOutputBuffers getOutputBuffers}</td>
1152 <td>[⁕↩]</td>
1164 <td class=fn>{@link #getOutputFormat()}</td>
1182 <td class=fn>{@link #getOutputFormat(int)}</td>
1200 <td class=fn>{@link #getOutputImage getOutputImage}</td>
1218 <td class=fn>{@link #queueInputBuffer queueInputBuffer}</td>
1236 <td class=fn>{@link #queueSecureInputBuffer queueSecureInputBuffer}</td>
1254 <td class=fn>{@link #release release}</td>
1272 <td class=fn>{@link #releaseOutputBuffer(int, boolean)}</td>
1290 <td class=fn>{@link #releaseOutputBuffer(int, long)}</td>
1308 <td class=fn>{@link #reset reset}</td>
1326 <td class=fn>{@link #setCallback(Callback) setCallback}</td>
1334 <td>{@link #setCallback(Callback, Handler) ⁕}</td>
1344 <td class=fn>{@link #setInputSurface setInputSurface}</td>
1362 <td class=fn>{@link #setOnFrameRenderedListener setOnFrameRenderedListener}</td>
1370 <td>○ ⎆</td>
1380 <td class=fn>{@link #setOutputSurface setOutputSurface}</td>
1398 <td class=fn>{@link #setParameters setParameters}</td>
1416 <td class=fn>{@link #setVideoScalingMode setVideoScalingMode}</td>
1434 <td class=fn>{@link #signalEndOfInputStream signalEndOfInputStream}</td>
1447 <td>21+(⇄)</td>
1452 <td class=fn>{@link #start start}</td>
1470 <td class=fn>{@link #stop stop}</td>
1483 final public class MediaCodec {
1485 * Per buffer metadata includes an offset and size specifying
1486 * the range of valid data in the associated codec (output) buffer.
1488 public final static class BufferInfo {
1490 * Update the buffer metadata information.
1492 * @param newOffset the start-offset of the data in the buffer.
1493 * @param newSize the amount of data (in bytes) in the buffer.
1494 * @param newTimeUs the presentation timestamp in microseconds.
1495 * @param newFlags buffer flags associated with the buffer. This
1496 * should be a combination of {@link #BUFFER_FLAG_KEY_FRAME} and
1497 * {@link #BUFFER_FLAG_END_OF_STREAM}.
1500 int newOffset, int newSize, long newTimeUs, @BufferFlag int newFlags) {
1503 presentationTimeUs = newTimeUs;
1508 * The start-offset of the data in the buffer.
1513 * The amount of data (in bytes) in the buffer. If this is {@code 0},
1514 * the buffer has no data in it and can be discarded. The only
1515 * use of a 0-size buffer is to carry the end-of-stream marker.
1520 * The presentation timestamp in microseconds for the buffer.
1521 * This is derived from the presentation timestamp passed in
1522 * with the corresponding input buffer. This should be ignored for
1525 public long presentationTimeUs;
1528 * Buffer flags associated with the buffer. A combination of
1529 * {@link #BUFFER_FLAG_KEY_FRAME} and {@link #BUFFER_FLAG_END_OF_STREAM}.
1531 * <p>Encoded buffers that are key frames are marked with
1532 * {@link #BUFFER_FLAG_KEY_FRAME}.
1534 * <p>The last output buffer corresponding to the input buffer
1535 * marked with {@link #BUFFER_FLAG_END_OF_STREAM} will also be marked
1536 * with {@link #BUFFER_FLAG_END_OF_STREAM}. In some cases this could
1537 * be an empty buffer, whose sole purpose is to carry the end-of-stream
1545 public BufferInfo dup() {
1546 BufferInfo copy = new BufferInfo();
1547 copy.set(offset, size, presentationTimeUs, flags);
1552 // The follow flag constants MUST stay in sync with their equivalents
1553 // in MediaCodec.h !
1556 * This indicates that the (encoded) buffer marked as such contains
1557 * the data for a key frame.
1559 * @deprecated Use {@link #BUFFER_FLAG_KEY_FRAME} instead.
1561 public static final int BUFFER_FLAG_SYNC_FRAME = 1;
1564 * This indicates that the (encoded) buffer marked as such contains
1565 * the data for a key frame.
1567 public static final int BUFFER_FLAG_KEY_FRAME = 1;
1570 * This indicated that the buffer marked as such contains codec
1571 * initialization / codec specific data instead of media data.
1573 public static final int BUFFER_FLAG_CODEC_CONFIG = 2;
1576 * This signals the end of stream, i.e. no buffers will be available
1577 * after this, unless of course, {@link #flush} follows.
1579 public static final int BUFFER_FLAG_END_OF_STREAM = 4;
1582 * This indicates that the buffer only contains part of a frame,
1583 * and the decoder should batch the data until a buffer without
1584 * this flag appears before decoding the frame.
1586 public static final int BUFFER_FLAG_PARTIAL_FRAME = 8;
1589 * This indicates that the buffer contains non-media data for the
1592 * All muxer data should start with a FOURCC header that determines the type of data.
1594 * For example, when it contains Exif data sent to a MediaMuxer track of
1595 * {@link MediaFormat#MIMETYPE_IMAGE_ANDROID_HEIC} type, the data must start with
1596 * Exif header ("Exif\0\0"), followed by the TIFF header (See JEITA CP-3451C Section 4.5.2.)
1600 public static final int BUFFER_FLAG_MUXER_DATA = 16;
1606 BUFFER_FLAG_SYNC_FRAME,
1607 BUFFER_FLAG_KEY_FRAME,
1608 BUFFER_FLAG_CODEC_CONFIG,
1609 BUFFER_FLAG_END_OF_STREAM,
1610 BUFFER_FLAG_PARTIAL_FRAME,
1611 BUFFER_FLAG_MUXER_DATA,
1613 @Retention(RetentionPolicy.SOURCE)
1614 public @interface BufferFlag {}
1616 private EventHandler mEventHandler;
1617 private EventHandler mOnFrameRenderedHandler;
1618 private EventHandler mCallbackHandler;
1619 private Callback mCallback;
1620 private OnFrameRenderedListener mOnFrameRenderedListener;
1621 private final Object mListenerLock = new Object();
1622 private MediaCodecInfo mCodecInfo;
1623 private final Object mCodecInfoLock = new Object();
1625 private static final int EVENT_CALLBACK = 1;
1626 private static final int EVENT_SET_CALLBACK = 2;
1627 private static final int EVENT_FRAME_RENDERED = 3;
1629 private static final int CB_INPUT_AVAILABLE = 1;
1630 private static final int CB_OUTPUT_AVAILABLE = 2;
1631 private static final int CB_ERROR = 3;
1632 private static final int CB_OUTPUT_FORMAT_CHANGE = 4;
1634 private class EventHandler extends Handler {
1635 private MediaCodec mCodec;
1637 public EventHandler(@NonNull MediaCodec codec, @NonNull Looper looper) {
1643 public void handleMessage(@NonNull Message msg) {
1645 case EVENT_CALLBACK:
1647 handleCallback(msg);
1650 case EVENT_SET_CALLBACK:
1652 mCallback = (MediaCodec.Callback) msg.obj;
1655 case EVENT_FRAME_RENDERED:
1656 synchronized (mListenerLock) {
1657 Map<String, Object> map = (Map<String, Object>)msg.obj;
1658 for (int i = 0; ; ++i) {
1659 Object mediaTimeUs = map.get(i + "-media-time-us");
1660 Object systemNano = map.get(i + "-system-nano");
1661 if (mediaTimeUs == null || systemNano == null
1662 || mOnFrameRenderedListener == null) {
1665 mOnFrameRenderedListener.onFrameRendered(
1666 mCodec, (long)mediaTimeUs, (long)systemNano);
1677 private void handleCallback(@NonNull Message msg) {
1678 if (mCallback == null) {
1683 case CB_INPUT_AVAILABLE:
1685 int index = msg.arg2;
1686 synchronized(mBufferLock) {
1687 validateInputByteBuffer(mCachedInputBuffers, index);
1689 mCallback.onInputBufferAvailable(mCodec, index);
1693 case CB_OUTPUT_AVAILABLE:
1695 int index = msg.arg2;
1696 BufferInfo info = (MediaCodec.BufferInfo) msg.obj;
1697 synchronized(mBufferLock) {
1698 validateOutputByteBuffer(mCachedOutputBuffers, index, info);
1700 mCallback.onOutputBufferAvailable(
1701 mCodec, index, info);
1707 mCallback.onError(mCodec, (MediaCodec.CodecException) msg.obj);
1711 case CB_OUTPUT_FORMAT_CHANGE:
1713 mCallback.onOutputFormatChanged(mCodec,
1714 new MediaFormat((Map<String, Object>) msg.obj));
1726 private boolean mHasSurface = false;
1729 * Instantiate the preferred decoder supporting input data of the given mime type.
1731 * The following is a partial list of defined mime types and their semantics:
1733 * <li>"video/x-vnd.on2.vp8" - VP8 video (i.e. video in .webm)
1734 * <li>"video/x-vnd.on2.vp9" - VP9 video (i.e. video in .webm)
1735 * <li>"video/avc" - H.264/AVC video
1736 * <li>"video/hevc" - H.265/HEVC video
1737 * <li>"video/mp4v-es" - MPEG4 video
1738 * <li>"video/3gpp" - H.263 video
1739 * <li>"audio/3gpp" - AMR narrowband audio
1740 * <li>"audio/amr-wb" - AMR wideband audio
1741 * <li>"audio/mpeg" - MPEG1/2 audio layer III
1742 * <li>"audio/mp4a-latm" - AAC audio (note, this is raw AAC packets, not packaged in LATM!)
1743 * <li>"audio/vorbis" - vorbis audio
1744 * <li>"audio/g711-alaw" - G.711 alaw audio
1745 * <li>"audio/g711-mlaw" - G.711 ulaw audio
1748 * <strong>Note:</strong> It is preferred to use {@link MediaCodecList#findDecoderForFormat}
1749 * and {@link #createByCodecName} to ensure that the resulting codec can handle a
1752 * @param type The mime type of the input data.
1753 * @throws IOException if the codec cannot be created.
1754 * @throws IllegalArgumentException if type is not a valid mime type.
1755 * @throws NullPointerException if type is null.
1758 public static MediaCodec createDecoderByType(@NonNull String type)
1759 throws IOException {
1760 return new MediaCodec(type, true /* nameIsType */, false /* encoder */);
1764 * Instantiate the preferred encoder supporting output data of the given mime type.
1766 * <strong>Note:</strong> It is preferred to use {@link MediaCodecList#findEncoderForFormat}
1767 * and {@link #createByCodecName} to ensure that the resulting codec can handle a
1770 * @param type The desired mime type of the output data.
1771 * @throws IOException if the codec cannot be created.
1772 * @throws IllegalArgumentException if type is not a valid mime type.
1773 * @throws NullPointerException if type is null.
1776 public static MediaCodec createEncoderByType(@NonNull String type)
1777 throws IOException {
1778 return new MediaCodec(type, true /* nameIsType */, true /* encoder */);
1782 * If you know the exact name of the component you want to instantiate
1783 * use this method to instantiate it. Use with caution.
1784 * Likely to be used with information obtained from {@link android.media.MediaCodecList}
1785 * @param name The name of the codec to be instantiated.
1786 * @throws IOException if the codec cannot be created.
1787 * @throws IllegalArgumentException if name is not valid.
1788 * @throws NullPointerException if name is null.
1791 public static MediaCodec createByCodecName(@NonNull String name)
1792 throws IOException {
1793 return new MediaCodec(
1794 name, false /* nameIsType */, false /* unused */);
1798 @NonNull String name, boolean nameIsType, boolean encoder) {
1800 if ((looper = Looper.myLooper()) != null) {
1801 mEventHandler = new EventHandler(this, looper);
1802 } else if ((looper = Looper.getMainLooper()) != null) {
1803 mEventHandler = new EventHandler(this, looper);
1805 mEventHandler = null;
1807 mCallbackHandler = mEventHandler;
1808 mOnFrameRenderedHandler = mEventHandler;
1810 mBufferLock = new Object();
1812 native_setup(name, nameIsType, encoder);
1816 protected void finalize() {
1821 * Returns the codec to its initial (Uninitialized) state.
1823 * Call this if an {@link MediaCodec.CodecException#isRecoverable unrecoverable}
1824 * error has occured to reset the codec to its initial state after creation.
1826 * @throws CodecException if an unrecoverable error has occured and the codec
1827 * could not be reset.
1828 * @throws IllegalStateException if in the Released state.
1830 public final void reset() {
1831 freeAllTrackedBuffers(); // free buffers first
1835 private native final void native_reset();
1838 * Free up resources used by the codec instance.
1840 * Make sure you call this when you're done to free up any opened
1841 * component instance instead of relying on the garbage collector
1842 * to do this for you at some point in the future.
1844 public final void release() {
1845 freeAllTrackedBuffers(); // free buffers first
1849 private native final void native_release();
1852 * If this codec is to be used as an encoder, pass this flag.
1854 public static final int CONFIGURE_FLAG_ENCODE = 1;
1857 @IntDef(flag = true, value = { CONFIGURE_FLAG_ENCODE })
1858 @Retention(RetentionPolicy.SOURCE)
1859 public @interface ConfigureFlag {}
1862 * Configures a component.
1864 * @param format The format of the input data (decoder) or the desired
1865 * format of the output data (encoder). Passing {@code null}
1866 * as {@code format} is equivalent to passing an
1867 * {@link MediaFormat#MediaFormat an empty mediaformat}.
1868 * @param surface Specify a surface on which to render the output of this
1869 * decoder. Pass {@code null} as {@code surface} if the
1870 * codec does not generate raw video output (e.g. not a video
1871 * decoder) and/or if you want to configure the codec for
1872 * {@link ByteBuffer} output.
1873 * @param crypto Specify a crypto object to facilitate secure decryption
1874 * of the media data. Pass {@code null} as {@code crypto} for
1875 * non-secure codecs.
1876 * @param flags Specify {@link #CONFIGURE_FLAG_ENCODE} to configure the
1877 * component as an encoder.
1878 * @throws IllegalArgumentException if the surface has been released (or is invalid),
1879 * or the format is unacceptable (e.g. missing a mandatory key),
1880 * or the flags are not set properly
1881 * (e.g. missing {@link #CONFIGURE_FLAG_ENCODE} for an encoder).
1882 * @throws IllegalStateException if not in the Uninitialized state.
1883 * @throws CryptoException upon DRM error.
1884 * @throws CodecException upon codec error.
1886 public void configure(
1887 @Nullable MediaFormat format,
1888 @Nullable Surface surface, @Nullable MediaCrypto crypto,
1889 @ConfigureFlag int flags) {
1890 configure(format, surface, crypto, null, flags);
1894 * Configure a component to be used with a descrambler.
1895 * @param format The format of the input data (decoder) or the desired
1896 * format of the output data (encoder). Passing {@code null}
1897 * as {@code format} is equivalent to passing an
1898 * {@link MediaFormat#MediaFormat an empty mediaformat}.
1899 * @param surface Specify a surface on which to render the output of this
1900 * decoder. Pass {@code null} as {@code surface} if the
1901 * codec does not generate raw video output (e.g. not a video
1902 * decoder) and/or if you want to configure the codec for
1903 * {@link ByteBuffer} output.
1904 * @param flags Specify {@link #CONFIGURE_FLAG_ENCODE} to configure the
1905 * component as an encoder.
1906 * @param descrambler Specify a descrambler object to facilitate secure
1907 * descrambling of the media data, or null for non-secure codecs.
1908 * @throws IllegalArgumentException if the surface has been released (or is invalid),
1909 * or the format is unacceptable (e.g. missing a mandatory key),
1910 * or the flags are not set properly
1911 * (e.g. missing {@link #CONFIGURE_FLAG_ENCODE} for an encoder).
1912 * @throws IllegalStateException if not in the Uninitialized state.
1913 * @throws CryptoException upon DRM error.
1914 * @throws CodecException upon codec error.
1916 public void configure(
1917 @Nullable MediaFormat format, @Nullable Surface surface,
1918 @ConfigureFlag int flags, @Nullable MediaDescrambler descrambler) {
1919 configure(format, surface, null,
1920 descrambler != null ? descrambler.getBinder() : null, flags);
1923 private void configure(
1924 @Nullable MediaFormat format, @Nullable Surface surface,
1925 @Nullable MediaCrypto crypto, @Nullable IHwBinder descramblerBinder,
1926 @ConfigureFlag int flags) {
1927 if (crypto != null && descramblerBinder != null) {
1928 throw new IllegalArgumentException("Can't use crypto and descrambler together!");
1931 String[] keys = null;
1932 Object[] values = null;
1934 if (format != null) {
1935 Map<String, Object> formatMap = format.getMap();
1936 keys = new String[formatMap.size()];
1937 values = new Object[formatMap.size()];
1940 for (Map.Entry<String, Object> entry: formatMap.entrySet()) {
1941 if (entry.getKey().equals(MediaFormat.KEY_AUDIO_SESSION_ID)) {
1944 sessionId = (Integer)entry.getValue();
1946 catch (Exception e) {
1947 throw new IllegalArgumentException("Wrong Session ID Parameter!");
1949 keys[i] = "audio-hw-sync";
1950 values[i] = AudioSystem.getAudioHwSyncForSession(sessionId);
1952 keys[i] = entry.getKey();
1953 values[i] = entry.getValue();
1959 mHasSurface = surface != null;
1961 native_configure(keys, values, surface, crypto, descramblerBinder, flags);
1965 * Dynamically sets the output surface of a codec.
1967 * This can only be used if the codec was configured with an output surface. The
1968 * new output surface should have a compatible usage type to the original output surface.
1969 * E.g. codecs may not support switching from a SurfaceTexture (GPU readable) output
1970 * to ImageReader (software readable) output.
1971 * @param surface the output surface to use. It must not be {@code null}.
1972 * @throws IllegalStateException if the codec does not support setting the output
1973 * surface in the current state.
1974 * @throws IllegalArgumentException if the new surface is not of a suitable type for the codec.
1976 public void setOutputSurface(@NonNull Surface surface) {
1978 throw new IllegalStateException("codec was not configured for an output surface");
1980 native_setSurface(surface);
1983 private native void native_setSurface(@NonNull Surface surface);
1986 * Create a persistent input surface that can be used with codecs that normally have an input
1987 * surface, such as video encoders. A persistent input can be reused by subsequent
1988 * {@link MediaCodec} or {@link MediaRecorder} instances, but can only be used by at
1989 * most one codec or recorder instance concurrently.
1991 * The application is responsible for calling release() on the Surface when done.
1993 * @return an input surface that can be used with {@link #setInputSurface}.
1996 public static Surface createPersistentInputSurface() {
1997 return native_createPersistentInputSurface();
2000 static class PersistentSurface extends Surface {
2001 @SuppressWarnings("unused")
2002 PersistentSurface() {} // used by native
2005 public void release() {
2006 native_releasePersistentInputSurface(this);
2010 private long mPersistentObject;
2014 * Configures the codec (e.g. encoder) to use a persistent input surface in place of input
2015 * buffers. This may only be called after {@link #configure} and before {@link #start}, in
2016 * lieu of {@link #createInputSurface}.
2017 * @param surface a persistent input surface created by {@link #createPersistentInputSurface}
2018 * @throws IllegalStateException if not in the Configured state or does not require an input
2020 * @throws IllegalArgumentException if the surface was not created by
2021 * {@link #createPersistentInputSurface}.
2023 public void setInputSurface(@NonNull Surface surface) {
2024 if (!(surface instanceof PersistentSurface)) {
2025 throw new IllegalArgumentException("not a PersistentSurface");
2027 native_setInputSurface(surface);
2031 private static native final PersistentSurface native_createPersistentInputSurface();
2032 private static native final void native_releasePersistentInputSurface(@NonNull Surface surface);
2033 private native final void native_setInputSurface(@NonNull Surface surface);
2035 private native final void native_setCallback(@Nullable Callback cb);
2037 private native final void native_configure(
2038 @Nullable String[] keys, @Nullable Object[] values,
2039 @Nullable Surface surface, @Nullable MediaCrypto crypto,
2040 @Nullable IHwBinder descramblerBinder, @ConfigureFlag int flags);
2043 * Requests a Surface to use as the input to an encoder, in place of input buffers. This
2044 * may only be called after {@link #configure} and before {@link #start}.
2046 * The application is responsible for calling release() on the Surface when
2049 * The Surface must be rendered with a hardware-accelerated API, such as OpenGL ES.
2050 * {@link android.view.Surface#lockCanvas(android.graphics.Rect)} may fail or produce
2051 * unexpected results.
2052 * @throws IllegalStateException if not in the Configured state.
2055 public native final Surface createInputSurface();
2058 * After successfully configuring the component, call {@code start}.
2060 * Call {@code start} also if the codec is configured in asynchronous mode,
2061 * and it has just been flushed, to resume requesting input buffers.
2062 * @throws IllegalStateException if not in the Configured state
2063 * or just after {@link #flush} for a codec that is configured
2064 * in asynchronous mode.
2065 * @throws MediaCodec.CodecException upon codec error. Note that some codec errors
2066 * for start may be attributed to future method calls.
2068 public final void start() {
2070 synchronized(mBufferLock) {
2071 cacheBuffers(true /* input */);
2072 cacheBuffers(false /* input */);
2075 private native final void native_start();
2078 * Finish the decode/encode session, note that the codec instance
2079 * remains active and ready to be {@link #start}ed again.
2080 * To ensure that it is available to other client call {@link #release}
2081 * and don't just rely on garbage collection to eventually do this for you.
2082 * @throws IllegalStateException if in the Released state.
2084 public final void stop() {
2086 freeAllTrackedBuffers();
2088 synchronized (mListenerLock) {
2089 if (mCallbackHandler != null) {
2090 mCallbackHandler.removeMessages(EVENT_SET_CALLBACK);
2091 mCallbackHandler.removeMessages(EVENT_CALLBACK);
2093 if (mOnFrameRenderedHandler != null) {
2094 mOnFrameRenderedHandler.removeMessages(EVENT_FRAME_RENDERED);
2099 private native final void native_stop();
2102 * Flush both input and output ports of the component.
2104 * Upon return, all indices previously returned in calls to {@link #dequeueInputBuffer
2105 * dequeueInputBuffer} and {@link #dequeueOutputBuffer dequeueOutputBuffer} — or obtained
2106 * via {@link Callback#onInputBufferAvailable onInputBufferAvailable} or
2107 * {@link Callback#onOutputBufferAvailable onOutputBufferAvailable} callbacks — become
2108 * invalid, and all buffers are owned by the codec.
2110 * If the codec is configured in asynchronous mode, call {@link #start}
2111 * after {@code flush} has returned to resume codec operations. The codec
2112 * will not request input buffers until this has happened.
2113 * <strong>Note, however, that there may still be outstanding {@code onOutputBufferAvailable}
2114 * callbacks that were not handled prior to calling {@code flush}.
2115 * The indices returned via these callbacks also become invalid upon calling {@code flush} and
2116 * should be discarded.</strong>
2118 * If the codec is configured in synchronous mode, codec will resume
2119 * automatically if it is configured with an input surface. Otherwise, it
2120 * will resume when {@link #dequeueInputBuffer dequeueInputBuffer} is called.
2122 * @throws IllegalStateException if not in the Executing state.
2123 * @throws MediaCodec.CodecException upon codec error.
2125 public final void flush() {
2126 synchronized(mBufferLock) {
2127 invalidateByteBuffers(mCachedInputBuffers);
2128 invalidateByteBuffers(mCachedOutputBuffers);
2129 mDequeuedInputBuffers.clear();
2130 mDequeuedOutputBuffers.clear();
2135 private native final void native_flush();
2138 * Thrown when an internal codec error occurs.
2140 public final static class CodecException extends IllegalStateException {
2141 CodecException(int errorCode, int actionCode, @Nullable String detailMessage) {
2142 super(detailMessage);
2143 mErrorCode = errorCode;
2144 mActionCode = actionCode;
2146 // TODO get this from codec
2147 final String sign = errorCode < 0 ? "neg_" : "";
2149 "android.media.MediaCodec.error_" + sign + Math.abs(errorCode);
2153 * Returns true if the codec exception is a transient issue,
2154 * perhaps due to resource constraints, and that the method
2155 * (or encoding/decoding) may be retried at a later time.
2157 public boolean isTransient() {
2158 return mActionCode == ACTION_TRANSIENT;
2162 * Returns true if the codec cannot proceed further,
2163 * but can be recovered by stopping, configuring,
2164 * and starting again.
2166 public boolean isRecoverable() {
2167 return mActionCode == ACTION_RECOVERABLE;
2171 * Retrieve the error code associated with a CodecException
2173 public int getErrorCode() {
2178 * Retrieve a developer-readable diagnostic information string
2179 * associated with the exception. Do not show this to end-users,
2180 * since this string will not be localized or generally
2181 * comprehensible to end-users.
2183 public @NonNull String getDiagnosticInfo() {
2184 return mDiagnosticInfo;
2188 * This indicates required resource was not able to be allocated.
2190 public static final int ERROR_INSUFFICIENT_RESOURCE = 1100;
2193 * This indicates the resource manager reclaimed the media resource used by the codec.
2195 * With this exception, the codec must be released, as it has moved to terminal state.
2197 public static final int ERROR_RECLAIMED = 1101;
2201 ERROR_INSUFFICIENT_RESOURCE,
2204 @Retention(RetentionPolicy.SOURCE)
2205 public @interface ReasonCode {}
2207 /* Must be in sync with android_media_MediaCodec.cpp */
2208 private final static int ACTION_TRANSIENT = 1;
2209 private final static int ACTION_RECOVERABLE = 2;
2211 private final String mDiagnosticInfo;
2212 private final int mErrorCode;
2213 private final int mActionCode;
2217 * Thrown when a crypto error occurs while queueing a secure input buffer.
2219 public final static class CryptoException extends RuntimeException {
2220 public CryptoException(int errorCode, @Nullable String detailMessage) {
2221 super(detailMessage);
2222 mErrorCode = errorCode;
2226 * This indicates that the requested key was not found when trying to
2227 * perform a decrypt operation. The operation can be retried after adding
2228 * the correct decryption key.
2230 public static final int ERROR_NO_KEY = 1;
2233 * This indicates that the key used for decryption is no longer
2234 * valid due to license term expiration. The operation can be retried
2235 * after updating the expired keys.
2237 public static final int ERROR_KEY_EXPIRED = 2;
2240 * This indicates that a required crypto resource was not able to be
2241 * allocated while attempting the requested operation. The operation
2242 * can be retried if the app is able to release resources.
2244 public static final int ERROR_RESOURCE_BUSY = 3;
2247 * This indicates that the output protection levels supported by the
2248 * device are not sufficient to meet the requirements set by the
2249 * content owner in the license policy.
2251 public static final int ERROR_INSUFFICIENT_OUTPUT_PROTECTION = 4;
2254 * This indicates that decryption was attempted on a session that is
2255 * not opened, which could be due to a failure to open the session,
2256 * closing the session prematurely, or the session being reclaimed
2257 * by the resource manager.
2259 public static final int ERROR_SESSION_NOT_OPENED = 5;
2262 * This indicates that an operation was attempted that could not be
2263 * supported by the crypto system of the device in its current
2264 * configuration. It may occur when the license policy requires
2265 * device security features that aren't supported by the device,
2266 * or due to an internal error in the crypto system that prevents
2267 * the specified security policy from being met.
2269 public static final int ERROR_UNSUPPORTED_OPERATION = 6;
2275 ERROR_RESOURCE_BUSY,
2276 ERROR_INSUFFICIENT_OUTPUT_PROTECTION,
2277 ERROR_SESSION_NOT_OPENED,
2278 ERROR_UNSUPPORTED_OPERATION
2280 @Retention(RetentionPolicy.SOURCE)
2281 public @interface CryptoErrorCode {}
2284 * Retrieve the error code associated with a CryptoException
2287 public int getErrorCode() {
2291 private int mErrorCode;
2295 * After filling a range of the input buffer at the specified index
2296 * submit it to the component. Once an input buffer is queued to
2297 * the codec, it MUST NOT be used until it is later retrieved by
2298 * {@link #getInputBuffer} in response to a {@link #dequeueInputBuffer}
2299 * return value or a {@link Callback#onInputBufferAvailable}
2302 * Many decoders require the actual compressed data stream to be
2303 * preceded by "codec specific data", i.e. setup data used to initialize
2304 * the codec such as PPS/SPS in the case of AVC video or code tables
2305 * in the case of vorbis audio.
2306 * The class {@link android.media.MediaExtractor} provides codec
2307 * specific data as part of
2308 * the returned track format in entries named "csd-0", "csd-1" ...
2310 * These buffers can be submitted directly after {@link #start} or
2311 * {@link #flush} by specifying the flag {@link
2312 * #BUFFER_FLAG_CODEC_CONFIG}. However, if you configure the
2313 * codec with a {@link MediaFormat} containing these keys, they
2314 * will be automatically submitted by MediaCodec directly after
2315 * start. Therefore, the use of {@link
2316 * #BUFFER_FLAG_CODEC_CONFIG} flag is discouraged and is
2317 * recommended only for advanced users.
2319 * To indicate that this is the final piece of input data (or rather that
2320 * no more input data follows unless the decoder is subsequently flushed)
2321 * specify the flag {@link #BUFFER_FLAG_END_OF_STREAM}.
2323 * <strong>Note:</strong> Prior to {@link android.os.Build.VERSION_CODES#M},
2324 * {@code presentationTimeUs} was not propagated to the frame timestamp of (rendered)
2325 * Surface output buffers, and the resulting frame timestamp was undefined.
2326 * Use {@link #releaseOutputBuffer(int, long)} to ensure a specific frame timestamp is set.
2327 * Similarly, since frame timestamps can be used by the destination surface for rendering
2328 * synchronization, <strong>care must be taken to normalize presentationTimeUs so as to not be
2329 * mistaken for a system time. (See {@linkplain #releaseOutputBuffer(int, long)
2330 * SurfaceView specifics}).</strong>
2332 * @param index The index of a client-owned input buffer previously returned
2333 * in a call to {@link #dequeueInputBuffer}.
2334 * @param offset The byte offset into the input buffer at which the data starts.
2335 * @param size The number of bytes of valid input data.
2336 * @param presentationTimeUs The presentation timestamp in microseconds for this
2337 * buffer. This is normally the media time at which this
2338 * buffer should be presented (rendered). When using an output
2339 * surface, this will be propagated as the {@link
2340 * SurfaceTexture#getTimestamp timestamp} for the frame (after
2341 * conversion to nanoseconds).
2342 * @param flags A bitmask of flags
2343 * {@link #BUFFER_FLAG_CODEC_CONFIG} and {@link #BUFFER_FLAG_END_OF_STREAM}.
2344 * While not prohibited, most codecs do not use the
2345 * {@link #BUFFER_FLAG_KEY_FRAME} flag for input buffers.
2346 * @throws IllegalStateException if not in the Executing state.
2347 * @throws MediaCodec.CodecException upon codec error.
2348 * @throws CryptoException if a crypto object has been specified in
2349 * {@link #configure}
2351 public final void queueInputBuffer(
2353 int offset, int size, long presentationTimeUs, int flags)
2354 throws CryptoException {
2355 synchronized(mBufferLock) {
2356 invalidateByteBuffer(mCachedInputBuffers, index);
2357 mDequeuedInputBuffers.remove(index);
2360 native_queueInputBuffer(
2361 index, offset, size, presentationTimeUs, flags);
2362 } catch (CryptoException | IllegalStateException e) {
2363 revalidateByteBuffer(mCachedInputBuffers, index);
2368 private native final void native_queueInputBuffer(
2370 int offset, int size, long presentationTimeUs, int flags)
2371 throws CryptoException;
2373 public static final int CRYPTO_MODE_UNENCRYPTED = 0;
2374 public static final int CRYPTO_MODE_AES_CTR = 1;
2375 public static final int CRYPTO_MODE_AES_CBC = 2;
2378 * Metadata describing the structure of an encrypted input sample.
2380 * A buffer's data is considered to be partitioned into "subSamples". Each subSample starts with
2381 * a run of plain, unencrypted bytes followed by a run of encrypted bytes. Either of these runs
2382 * may be empty. If pattern encryption applies, each of the encrypted runs is encrypted only
2383 * partly, according to a repeating pattern of "encrypt" and "skip" blocks.
2384 * {@link #numBytesOfClearData} can be null to indicate that all data is encrypted, and
2385 * {@link #numBytesOfEncryptedData} can be null to indicate that all data is clear. At least one
2386 * of {@link #numBytesOfClearData} and {@link #numBytesOfEncryptedData} must be non-null.
2388 * This information encapsulates per-sample metadata as outlined in ISO/IEC FDIS 23001-7:2016
2389 * "Common encryption in ISO base media file format files".
2391 * <h3>ISO-CENC Schemes</h3>
2392 * ISO/IEC FDIS 23001-7:2016 defines four possible schemes by which media may be encrypted,
2393 * corresponding to each possible combination of an AES mode with the presence or absence of
2394 * patterned encryption.
2396 * <table style="width: 0%">
2406 * <th>Without Patterns</th>
2410 * <th>With Patterns</th>
2417 * For {@code CryptoInfo}, the scheme is selected implicitly by the combination of the
2418 * {@link #mode} field and the value set with {@link #setPattern}. For the pattern, setting the
2419 * pattern to all zeroes (that is, both {@code blocksToEncrypt} and {@code blocksToSkip} are
2420 * zero) is interpreted as turning patterns off completely. A scheme that does not use patterns
2421 * will be selected, either cenc or cbc1. Setting the pattern to any nonzero value will choose
2422 * one of the pattern-supporting schemes, cens or cbcs. The default pattern if
2423 * {@link #setPattern} is never called is all zeroes.
2425 * <h4>HLS SAMPLE-AES Audio</h4>
2426 * HLS SAMPLE-AES audio is encrypted in a manner compatible with the cbcs scheme, except that it
2427 * does not use patterned encryption. However, if {@link #setPattern} is used to set the pattern
2428 * to all zeroes, this will be interpreted as selecting the cbc1 scheme. The cbc1 scheme cannot
2429 * successfully decrypt HLS SAMPLE-AES audio because of differences in how the IVs are handled.
2430 * For this reason, it is recommended that a pattern of {@code 1} encrypted block and {@code 0}
2431 * skip blocks be used with HLS SAMPLE-AES audio. This will trigger decryption to use cbcs mode
2432 * while still decrypting every block.
2434 public final static class CryptoInfo {
2436 * The number of subSamples that make up the buffer's contents.
2438 public int numSubSamples;
2440 * The number of leading unencrypted bytes in each subSample. If null, all bytes are treated
2441 * as encrypted and {@link #numBytesOfEncryptedData} must be specified.
2443 public int[] numBytesOfClearData;
2445 * The number of trailing encrypted bytes in each subSample. If null, all bytes are treated
2446 * as clear and {@link #numBytesOfClearData} must be specified.
2448 public int[] numBytesOfEncryptedData;
2454 * A 16-byte initialization vector
2458 * The type of encryption that has been applied,
2459 * see {@link #CRYPTO_MODE_UNENCRYPTED}, {@link #CRYPTO_MODE_AES_CTR}
2460 * and {@link #CRYPTO_MODE_AES_CBC}
2465 * Metadata describing an encryption pattern for the protected bytes in a subsample. An
2466 * encryption pattern consists of a repeating sequence of crypto blocks comprised of a
2467 * number of encrypted blocks followed by a number of unencrypted, or skipped, blocks.
2469 public final static class Pattern {
2471 * Number of blocks to be encrypted in the pattern. If both this and
2472 * {@link #mSkipBlocks} are zero, pattern encryption is inoperative.
2474 private int mEncryptBlocks;
2477 * Number of blocks to be skipped (left clear) in the pattern. If both this and
2478 * {@link #mEncryptBlocks} are zero, pattern encryption is inoperative.
2480 private int mSkipBlocks;
2483 * Construct a sample encryption pattern given the number of blocks to encrypt and skip
2484 * in the pattern. If both parameters are zero, pattern encryption is inoperative.
2486 public Pattern(int blocksToEncrypt, int blocksToSkip) {
2487 set(blocksToEncrypt, blocksToSkip);
2491 * Set the number of blocks to encrypt and skip in a sample encryption pattern. If both
2492 * parameters are zero, pattern encryption is inoperative.
2494 public void set(int blocksToEncrypt, int blocksToSkip) {
2495 mEncryptBlocks = blocksToEncrypt;
2496 mSkipBlocks = blocksToSkip;
2500 * Return the number of blocks to skip in a sample encryption pattern.
2502 public int getSkipBlocks() {
2507 * Return the number of blocks to encrypt in a sample encryption pattern.
2509 public int getEncryptBlocks() {
2510 return mEncryptBlocks;
2514 private final Pattern zeroPattern = new Pattern(0, 0);
2517 * The pattern applicable to the protected data in each subsample.
2519 private Pattern pattern;
2522 * Set the subsample count, clear/encrypted sizes, key, IV and mode fields of
2523 * a {@link MediaCodec.CryptoInfo} instance.
2526 int newNumSubSamples,
2527 @NonNull int[] newNumBytesOfClearData,
2528 @NonNull int[] newNumBytesOfEncryptedData,
2529 @NonNull byte[] newKey,
2530 @NonNull byte[] newIV,
2532 numSubSamples = newNumSubSamples;
2533 numBytesOfClearData = newNumBytesOfClearData;
2534 numBytesOfEncryptedData = newNumBytesOfEncryptedData;
2538 pattern = zeroPattern;
2542 * Set the encryption pattern on a {@link MediaCodec.CryptoInfo} instance.
2543 * See {@link MediaCodec.CryptoInfo.Pattern}.
2545 public void setPattern(Pattern newPattern) {
2546 pattern = newPattern;
2549 private void setPattern(int blocksToEncrypt, int blocksToSkip) {
2550 pattern = new Pattern(blocksToEncrypt, blocksToSkip);
2554 public String toString() {
2555 StringBuilder builder = new StringBuilder();
2556 builder.append(numSubSamples + " subsamples, key [");
2557 String hexdigits = "0123456789abcdef";
2558 for (int i = 0; i < key.length; i++) {
2559 builder.append(hexdigits.charAt((key[i] & 0xf0) >> 4));
2560 builder.append(hexdigits.charAt(key[i] & 0x0f));
2562 builder.append("], iv [");
2563 for (int i = 0; i < key.length; i++) {
2564 builder.append(hexdigits.charAt((iv[i] & 0xf0) >> 4));
2565 builder.append(hexdigits.charAt(iv[i] & 0x0f));
2567 builder.append("], clear ");
2568 builder.append(Arrays.toString(numBytesOfClearData));
2569 builder.append(", encrypted ");
2570 builder.append(Arrays.toString(numBytesOfEncryptedData));
2571 return builder.toString();
2576 * Similar to {@link #queueInputBuffer queueInputBuffer} but submits a buffer that is
2577 * potentially encrypted.
2578 * <strong>Check out further notes at {@link #queueInputBuffer queueInputBuffer}.</strong>
2580 * @param index The index of a client-owned input buffer previously returned
2581 * in a call to {@link #dequeueInputBuffer}.
2582 * @param offset The byte offset into the input buffer at which the data starts.
2583 * @param info Metadata required to facilitate decryption, the object can be
2584 * reused immediately after this call returns.
2585 * @param presentationTimeUs The presentation timestamp in microseconds for this
2586 * buffer. This is normally the media time at which this
2587 * buffer should be presented (rendered).
2588 * @param flags A bitmask of flags
2589 * {@link #BUFFER_FLAG_CODEC_CONFIG} and {@link #BUFFER_FLAG_END_OF_STREAM}.
2590 * While not prohibited, most codecs do not use the
2591 * {@link #BUFFER_FLAG_KEY_FRAME} flag for input buffers.
2592 * @throws IllegalStateException if not in the Executing state.
2593 * @throws MediaCodec.CodecException upon codec error.
2594 * @throws CryptoException if an error occurs while attempting to decrypt the buffer.
2595 * An error code associated with the exception helps identify the
2596 * reason for the failure.
2598 public final void queueSecureInputBuffer(
2601 @NonNull CryptoInfo info,
2602 long presentationTimeUs,
2603 int flags) throws CryptoException {
2604 synchronized(mBufferLock) {
2605 invalidateByteBuffer(mCachedInputBuffers, index);
2606 mDequeuedInputBuffers.remove(index);
2609 native_queueSecureInputBuffer(
2610 index, offset, info, presentationTimeUs, flags);
2611 } catch (CryptoException | IllegalStateException e) {
2612 revalidateByteBuffer(mCachedInputBuffers, index);
2617 private native final void native_queueSecureInputBuffer(
2620 @NonNull CryptoInfo info,
2621 long presentationTimeUs,
2622 int flags) throws CryptoException;
2625 * Returns the index of an input buffer to be filled with valid data
2626 * or -1 if no such buffer is currently available.
2627 * This method will return immediately if timeoutUs == 0, wait indefinitely
2628 * for the availability of an input buffer if timeoutUs < 0 or wait up
2629 * to "timeoutUs" microseconds if timeoutUs > 0.
2630 * @param timeoutUs The timeout in microseconds, a negative timeout indicates "infinite".
2631 * @throws IllegalStateException if not in the Executing state,
2632 * or codec is configured in asynchronous mode.
2633 * @throws MediaCodec.CodecException upon codec error.
2635 public final int dequeueInputBuffer(long timeoutUs) {
2636 int res = native_dequeueInputBuffer(timeoutUs);
2638 synchronized(mBufferLock) {
2639 validateInputByteBuffer(mCachedInputBuffers, res);
2645 private native final int native_dequeueInputBuffer(long timeoutUs);
2648 * If a non-negative timeout had been specified in the call
2649 * to {@link #dequeueOutputBuffer}, indicates that the call timed out.
2651 public static final int INFO_TRY_AGAIN_LATER = -1;
2654 * The output format has changed, subsequent data will follow the new
2655 * format. {@link #getOutputFormat()} returns the new format. Note, that
2656 * you can also use the new {@link #getOutputFormat(int)} method to
2657 * get the format for a specific output buffer. This frees you from
2658 * having to track output format changes.
2660 public static final int INFO_OUTPUT_FORMAT_CHANGED = -2;
2663 * The output buffers have changed, the client must refer to the new
2664 * set of output buffers returned by {@link #getOutputBuffers} from
2667 * <p>Additionally, this event signals that the video scaling mode
2668 * may have been reset to the default.</p>
2670 * @deprecated This return value can be ignored as {@link
2671 * #getOutputBuffers} has been deprecated. Client should
2672 * request a current buffer using on of the get-buffer or
2673 * get-image methods each time one has been dequeued.
2675 public static final int INFO_OUTPUT_BUFFERS_CHANGED = -3;
2679 INFO_TRY_AGAIN_LATER,
2680 INFO_OUTPUT_FORMAT_CHANGED,
2681 INFO_OUTPUT_BUFFERS_CHANGED,
2683 @Retention(RetentionPolicy.SOURCE)
2684 public @interface OutputBufferInfo {}
2687 * Dequeue an output buffer, block at most "timeoutUs" microseconds.
2688 * Returns the index of an output buffer that has been successfully
2689 * decoded or one of the INFO_* constants.
2690 * @param info Will be filled with buffer meta data.
2691 * @param timeoutUs The timeout in microseconds, a negative timeout indicates "infinite".
2692 * @throws IllegalStateException if not in the Executing state,
2693 * or codec is configured in asynchronous mode.
2694 * @throws MediaCodec.CodecException upon codec error.
2697 public final int dequeueOutputBuffer(
2698 @NonNull BufferInfo info, long timeoutUs) {
2699 int res = native_dequeueOutputBuffer(info, timeoutUs);
2700 synchronized(mBufferLock) {
2701 if (res == INFO_OUTPUT_BUFFERS_CHANGED) {
2702 cacheBuffers(false /* input */);
2703 } else if (res >= 0) {
2704 validateOutputByteBuffer(mCachedOutputBuffers, res, info);
2706 mDequeuedOutputInfos.put(res, info.dup());
2713 private native final int native_dequeueOutputBuffer(
2714 @NonNull BufferInfo info, long timeoutUs);
2717 * If you are done with a buffer, use this call to return the buffer to the codec
2718 * or to render it on the output surface. If you configured the codec with an
2719 * output surface, setting {@code render} to {@code true} will first send the buffer
2720 * to that output surface. The surface will release the buffer back to the codec once
2721 * it is no longer used/displayed.
2723 * Once an output buffer is released to the codec, it MUST NOT
2724 * be used until it is later retrieved by {@link #getOutputBuffer} in response
2725 * to a {@link #dequeueOutputBuffer} return value or a
2726 * {@link Callback#onOutputBufferAvailable} callback.
2728 * @param index The index of a client-owned output buffer previously returned
2729 * from a call to {@link #dequeueOutputBuffer}.
2730 * @param render If a valid surface was specified when configuring the codec,
2731 * passing true renders this output buffer to the surface.
2732 * @throws IllegalStateException if not in the Executing state.
2733 * @throws MediaCodec.CodecException upon codec error.
2735 public final void releaseOutputBuffer(int index, boolean render) {
2736 BufferInfo info = null;
2737 synchronized(mBufferLock) {
2738 invalidateByteBuffer(mCachedOutputBuffers, index);
2739 mDequeuedOutputBuffers.remove(index);
2741 info = mDequeuedOutputInfos.remove(index);
2744 releaseOutputBuffer(index, render, false /* updatePTS */, 0 /* dummy */);
2748 * If you are done with a buffer, use this call to update its surface timestamp
2749 * and return it to the codec to render it on the output surface. If you
2750 * have not specified an output surface when configuring this video codec,
2751 * this call will simply return the buffer to the codec.<p>
2753 * The timestamp may have special meaning depending on the destination surface.
2756 * <tr><th>SurfaceView specifics</th></tr>
2758 * If you render your buffer on a {@link android.view.SurfaceView},
2759 * you can use the timestamp to render the buffer at a specific time (at the
2760 * VSYNC at or after the buffer timestamp). For this to work, the timestamp
2761 * needs to be <i>reasonably close</i> to the current {@link System#nanoTime}.
2762 * Currently, this is set as within one (1) second. A few notes:
2765 * <li>the buffer will not be returned to the codec until the timestamp
2766 * has passed and the buffer is no longer used by the {@link android.view.Surface}.
2767 * <li>buffers are processed sequentially, so you may block subsequent buffers to
2768 * be displayed on the {@link android.view.Surface}. This is important if you
2769 * want to react to user action, e.g. stop the video or seek.
2770 * <li>if multiple buffers are sent to the {@link android.view.Surface} to be
2771 * rendered at the same VSYNC, the last one will be shown, and the other ones
2773 * <li>if the timestamp is <em>not</em> "reasonably close" to the current system
2774 * time, the {@link android.view.Surface} will ignore the timestamp, and
2775 * display the buffer at the earliest feasible time. In this mode it will not
2777 * <li>for best performance and quality, call this method when you are about
2778 * two VSYNCs' time before the desired render time. For 60Hz displays, this is
2784 * Once an output buffer is released to the codec, it MUST NOT
2785 * be used until it is later retrieved by {@link #getOutputBuffer} in response
2786 * to a {@link #dequeueOutputBuffer} return value or a
2787 * {@link Callback#onOutputBufferAvailable} callback.
2789 * @param index The index of a client-owned output buffer previously returned
2790 * from a call to {@link #dequeueOutputBuffer}.
2791 * @param renderTimestampNs The timestamp to associate with this buffer when
2792 * it is sent to the Surface.
2793 * @throws IllegalStateException if not in the Executing state.
2794 * @throws MediaCodec.CodecException upon codec error.
2796 public final void releaseOutputBuffer(int index, long renderTimestampNs) {
2797 BufferInfo info = null;
2798 synchronized(mBufferLock) {
2799 invalidateByteBuffer(mCachedOutputBuffers, index);
2800 mDequeuedOutputBuffers.remove(index);
2802 info = mDequeuedOutputInfos.remove(index);
2805 releaseOutputBuffer(
2806 index, true /* render */, true /* updatePTS */, renderTimestampNs);
2809 private native final void releaseOutputBuffer(
2810 int index, boolean render, boolean updatePTS, long timeNs);
2813 * Signals end-of-stream on input. Equivalent to submitting an empty buffer with
2814 * {@link #BUFFER_FLAG_END_OF_STREAM} set. This may only be used with
2815 * encoders receiving input from a Surface created by {@link #createInputSurface}.
2816 * @throws IllegalStateException if not in the Executing state.
2817 * @throws MediaCodec.CodecException upon codec error.
2819 public native final void signalEndOfInputStream();
2822 * Call this after dequeueOutputBuffer signals a format change by returning
2823 * {@link #INFO_OUTPUT_FORMAT_CHANGED}.
2824 * You can also call this after {@link #configure} returns
2825 * successfully to get the output format initially configured
2826 * for the codec. Do this to determine what optional
2827 * configuration parameters were supported by the codec.
2829 * @throws IllegalStateException if not in the Executing or
2831 * @throws MediaCodec.CodecException upon codec error.
2834 public final MediaFormat getOutputFormat() {
2835 return new MediaFormat(getFormatNative(false /* input */));
2839 * Call this after {@link #configure} returns successfully to
2840 * get the input format accepted by the codec. Do this to
2841 * determine what optional configuration parameters were
2842 * supported by the codec.
2844 * @throws IllegalStateException if not in the Executing or
2846 * @throws MediaCodec.CodecException upon codec error.
2849 public final MediaFormat getInputFormat() {
2850 return new MediaFormat(getFormatNative(true /* input */));
2854 * Returns the output format for a specific output buffer.
2856 * @param index The index of a client-owned input buffer previously
2857 * returned from a call to {@link #dequeueInputBuffer}.
2859 * @return the format for the output buffer, or null if the index
2860 * is not a dequeued output buffer.
2863 public final MediaFormat getOutputFormat(int index) {
2864 return new MediaFormat(getOutputFormatNative(index));
2868 private native final Map<String, Object> getFormatNative(boolean input);
2871 private native final Map<String, Object> getOutputFormatNative(int index);
2873 // used to track dequeued buffers
2874 private static class BufferMap {
2875 // various returned representations of the codec buffer
2876 private static class CodecBuffer {
2877 private Image mImage;
2878 private ByteBuffer mByteBuffer;
2880 public void free() {
2881 if (mByteBuffer != null) {
2882 // all of our ByteBuffers are direct
2883 java.nio.NioUtils.freeDirectBuffer(mByteBuffer);
2886 if (mImage != null) {
2892 public void setImage(@Nullable Image image) {
2897 public void setByteBuffer(@Nullable ByteBuffer buffer) {
2899 mByteBuffer = buffer;
2903 private final Map<Integer, CodecBuffer> mMap =
2904 new HashMap<Integer, CodecBuffer>();
2906 public void remove(int index) {
2907 CodecBuffer buffer = mMap.get(index);
2908 if (buffer != null) {
2914 public void put(int index, @Nullable ByteBuffer newBuffer) {
2915 CodecBuffer buffer = mMap.get(index);
2916 if (buffer == null) { // likely
2917 buffer = new CodecBuffer();
2918 mMap.put(index, buffer);
2920 buffer.setByteBuffer(newBuffer);
2923 public void put(int index, @Nullable Image newImage) {
2924 CodecBuffer buffer = mMap.get(index);
2925 if (buffer == null) { // likely
2926 buffer = new CodecBuffer();
2927 mMap.put(index, buffer);
2929 buffer.setImage(newImage);
2932 public void clear() {
2933 for (CodecBuffer buffer: mMap.values()) {
2940 private ByteBuffer[] mCachedInputBuffers;
2941 private ByteBuffer[] mCachedOutputBuffers;
2942 private final BufferMap mDequeuedInputBuffers = new BufferMap();
2943 private final BufferMap mDequeuedOutputBuffers = new BufferMap();
2944 private final Map<Integer, BufferInfo> mDequeuedOutputInfos =
2945 new HashMap<Integer, BufferInfo>();
2946 final private Object mBufferLock;
2948 private final void invalidateByteBuffer(
2949 @Nullable ByteBuffer[] buffers, int index) {
2950 if (buffers != null && index >= 0 && index < buffers.length) {
2951 ByteBuffer buffer = buffers[index];
2952 if (buffer != null) {
2953 buffer.setAccessible(false);
2958 private final void validateInputByteBuffer(
2959 @Nullable ByteBuffer[] buffers, int index) {
2960 if (buffers != null && index >= 0 && index < buffers.length) {
2961 ByteBuffer buffer = buffers[index];
2962 if (buffer != null) {
2963 buffer.setAccessible(true);
2969 private final void revalidateByteBuffer(
2970 @Nullable ByteBuffer[] buffers, int index) {
2971 synchronized(mBufferLock) {
2972 if (buffers != null && index >= 0 && index < buffers.length) {
2973 ByteBuffer buffer = buffers[index];
2974 if (buffer != null) {
2975 buffer.setAccessible(true);
2981 private final void validateOutputByteBuffer(
2982 @Nullable ByteBuffer[] buffers, int index, @NonNull BufferInfo info) {
2983 if (buffers != null && index >= 0 && index < buffers.length) {
2984 ByteBuffer buffer = buffers[index];
2985 if (buffer != null) {
2986 buffer.setAccessible(true);
2987 buffer.limit(info.offset + info.size).position(info.offset);
2992 private final void invalidateByteBuffers(@Nullable ByteBuffer[] buffers) {
2993 if (buffers != null) {
2994 for (ByteBuffer buffer: buffers) {
2995 if (buffer != null) {
2996 buffer.setAccessible(false);
3002 private final void freeByteBuffer(@Nullable ByteBuffer buffer) {
3003 if (buffer != null /* && buffer.isDirect() */) {
3004 // all of our ByteBuffers are direct
3005 java.nio.NioUtils.freeDirectBuffer(buffer);
3009 private final void freeByteBuffers(@Nullable ByteBuffer[] buffers) {
3010 if (buffers != null) {
3011 for (ByteBuffer buffer: buffers) {
3012 freeByteBuffer(buffer);
3017 private final void freeAllTrackedBuffers() {
3018 synchronized(mBufferLock) {
3019 freeByteBuffers(mCachedInputBuffers);
3020 freeByteBuffers(mCachedOutputBuffers);
3021 mCachedInputBuffers = null;
3022 mCachedOutputBuffers = null;
3023 mDequeuedInputBuffers.clear();
3024 mDequeuedOutputBuffers.clear();
3028 private final void cacheBuffers(boolean input) {
3029 ByteBuffer[] buffers = null;
3031 buffers = getBuffers(input);
3032 invalidateByteBuffers(buffers);
3033 } catch (IllegalStateException e) {
3034 // we don't get buffers in async mode
3037 mCachedInputBuffers = buffers;
3039 mCachedOutputBuffers = buffers;
3044 * Retrieve the set of input buffers. Call this after start()
3045 * returns. After calling this method, any ByteBuffers
3046 * previously returned by an earlier call to this method MUST no
3049 * @deprecated Use the new {@link #getInputBuffer} method instead
3050 * each time an input buffer is dequeued.
3052 * <b>Note:</b> As of API 21, dequeued input buffers are
3053 * automatically {@link java.nio.Buffer#clear cleared}.
3055 * <em>Do not use this method if using an input surface.</em>
3057 * @throws IllegalStateException if not in the Executing state,
3058 * or codec is configured in asynchronous mode.
3059 * @throws MediaCodec.CodecException upon codec error.
3062 public ByteBuffer[] getInputBuffers() {
3063 if (mCachedInputBuffers == null) {
3064 throw new IllegalStateException();
3066 // FIXME: check codec status
3067 return mCachedInputBuffers;
3071 * Retrieve the set of output buffers. Call this after start()
3072 * returns and whenever dequeueOutputBuffer signals an output
3073 * buffer change by returning {@link
3074 * #INFO_OUTPUT_BUFFERS_CHANGED}. After calling this method, any
3075 * ByteBuffers previously returned by an earlier call to this
3076 * method MUST no longer be used.
3078 * @deprecated Use the new {@link #getOutputBuffer} method instead
3079 * each time an output buffer is dequeued. This method is not
3080 * supported if codec is configured in asynchronous mode.
3082 * <b>Note:</b> As of API 21, the position and limit of output
3083 * buffers that are dequeued will be set to the valid data
3086 * <em>Do not use this method if using an output surface.</em>
3088 * @throws IllegalStateException if not in the Executing state,
3089 * or codec is configured in asynchronous mode.
3090 * @throws MediaCodec.CodecException upon codec error.
3093 public ByteBuffer[] getOutputBuffers() {
3094 if (mCachedOutputBuffers == null) {
3095 throw new IllegalStateException();
3097 // FIXME: check codec status
3098 return mCachedOutputBuffers;
3102 * Returns a {@link java.nio.Buffer#clear cleared}, writable ByteBuffer
3103 * object for a dequeued input buffer index to contain the input data.
3105 * After calling this method any ByteBuffer or Image object
3106 * previously returned for the same input index MUST no longer
3109 * @param index The index of a client-owned input buffer previously
3110 * returned from a call to {@link #dequeueInputBuffer},
3111 * or received via an onInputBufferAvailable callback.
3113 * @return the input buffer, or null if the index is not a dequeued
3114 * input buffer, or if the codec is configured for surface input.
3116 * @throws IllegalStateException if not in the Executing state.
3117 * @throws MediaCodec.CodecException upon codec error.
3120 public ByteBuffer getInputBuffer(int index) {
3121 ByteBuffer newBuffer = getBuffer(true /* input */, index);
3122 synchronized(mBufferLock) {
3123 invalidateByteBuffer(mCachedInputBuffers, index);
3124 mDequeuedInputBuffers.put(index, newBuffer);
3130 * Returns a writable Image object for a dequeued input buffer
3131 * index to contain the raw input video frame.
3133 * After calling this method any ByteBuffer or Image object
3134 * previously returned for the same input index MUST no longer
3137 * @param index The index of a client-owned input buffer previously
3138 * returned from a call to {@link #dequeueInputBuffer},
3139 * or received via an onInputBufferAvailable callback.
3141 * @return the input image, or null if the index is not a
3142 * dequeued input buffer, or not a ByteBuffer that contains a
3145 * @throws IllegalStateException if not in the Executing state.
3146 * @throws MediaCodec.CodecException upon codec error.
3149 public Image getInputImage(int index) {
3150 Image newImage = getImage(true /* input */, index);
3151 synchronized(mBufferLock) {
3152 invalidateByteBuffer(mCachedInputBuffers, index);
3153 mDequeuedInputBuffers.put(index, newImage);
3159 * Returns a read-only ByteBuffer for a dequeued output buffer
3160 * index. The position and limit of the returned buffer are set
3161 * to the valid output data.
3163 * After calling this method, any ByteBuffer or Image object
3164 * previously returned for the same output index MUST no longer
3167 * @param index The index of a client-owned output buffer previously
3168 * returned from a call to {@link #dequeueOutputBuffer},
3169 * or received via an onOutputBufferAvailable callback.
3171 * @return the output buffer, or null if the index is not a dequeued
3172 * output buffer, or the codec is configured with an output surface.
3174 * @throws IllegalStateException if not in the Executing state.
3175 * @throws MediaCodec.CodecException upon codec error.
3178 public ByteBuffer getOutputBuffer(int index) {
3179 ByteBuffer newBuffer = getBuffer(false /* input */, index);
3180 synchronized(mBufferLock) {
3181 invalidateByteBuffer(mCachedOutputBuffers, index);
3182 mDequeuedOutputBuffers.put(index, newBuffer);
3188 * Returns a read-only Image object for a dequeued output buffer
3189 * index that contains the raw video frame.
3191 * After calling this method, any ByteBuffer or Image object previously
3192 * returned for the same output index MUST no longer be used.
3194 * @param index The index of a client-owned output buffer previously
3195 * returned from a call to {@link #dequeueOutputBuffer},
3196 * or received via an onOutputBufferAvailable callback.
3198 * @return the output image, or null if the index is not a
3199 * dequeued output buffer, not a raw video frame, or if the codec
3200 * was configured with an output surface.
3202 * @throws IllegalStateException if not in the Executing state.
3203 * @throws MediaCodec.CodecException upon codec error.
3206 public Image getOutputImage(int index) {
3207 Image newImage = getImage(false /* input */, index);
3208 synchronized(mBufferLock) {
3209 invalidateByteBuffer(mCachedOutputBuffers, index);
3210 mDequeuedOutputBuffers.put(index, newImage);
3216 * The content is scaled to the surface dimensions
3218 public static final int VIDEO_SCALING_MODE_SCALE_TO_FIT = 1;
3221 * The content is scaled, maintaining its aspect ratio, the whole
3222 * surface area is used, content may be cropped.
3224 * This mode is only suitable for content with 1:1 pixel aspect ratio as you cannot
3225 * configure the pixel aspect ratio for a {@link Surface}.
3227 * As of {@link android.os.Build.VERSION_CODES#N} release, this mode may not work if
3228 * the video is {@linkplain MediaFormat#KEY_ROTATION rotated} by 90 or 270 degrees.
3230 public static final int VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING = 2;
3234 VIDEO_SCALING_MODE_SCALE_TO_FIT,
3235 VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING,
3237 @Retention(RetentionPolicy.SOURCE)
3238 public @interface VideoScalingMode {}
3241 * If a surface has been specified in a previous call to {@link #configure}
3242 * specifies the scaling mode to use. The default is "scale to fit".
3244 * The scaling mode may be reset to the <strong>default</strong> each time an
3245 * {@link #INFO_OUTPUT_BUFFERS_CHANGED} event is received from the codec; therefore, the client
3246 * must call this method after every buffer change event (and before the first output buffer is
3247 * released for rendering) to ensure consistent scaling mode.
3249 * Since the {@link #INFO_OUTPUT_BUFFERS_CHANGED} event is deprecated, this can also be done
3250 * after each {@link #INFO_OUTPUT_FORMAT_CHANGED} event.
3252 * @throws IllegalArgumentException if mode is not recognized.
3253 * @throws IllegalStateException if in the Released state.
3255 public native final void setVideoScalingMode(@VideoScalingMode int mode);
3258 * Get the component name. If the codec was created by createDecoderByType
3259 * or createEncoderByType, what component is chosen is not known beforehand.
3260 * @throws IllegalStateException if in the Released state.
3263 public native final String getName();
3266 * Return Metrics data about the current codec instance.
3268 * @return a {@link PersistableBundle} containing the set of attributes and values
3269 * available for the media being handled by this instance of MediaCodec
3270 * The attributes are descibed in {@link MetricsConstants}.
3272 * Additional vendor-specific fields may also be present in
3275 public PersistableBundle getMetrics() {
3276 PersistableBundle bundle = native_getMetrics();
3280 private native PersistableBundle native_getMetrics();
3283 * Change a video encoder's target bitrate on the fly. The value is an
3284 * Integer object containing the new bitrate in bps.
3286 public static final String PARAMETER_KEY_VIDEO_BITRATE = "video-bitrate";
3289 * Temporarily suspend/resume encoding of input data. While suspended
3290 * input data is effectively discarded instead of being fed into the
3291 * encoder. This parameter really only makes sense to use with an encoder
3292 * in "surface-input" mode, as the client code has no control over the
3293 * input-side of the encoder in that case.
3294 * The value is an Integer object containing the value 1 to suspend
3295 * or the value 0 to resume.
3297 public static final String PARAMETER_KEY_SUSPEND = "drop-input-frames";
3300 * Request that the encoder produce a sync frame "soon".
3301 * Provide an Integer with the value 0.
3303 public static final String PARAMETER_KEY_REQUEST_SYNC_FRAME = "request-sync";
3306 * Communicate additional parameter changes to the component instance.
3307 * <b>Note:</b> Some of these parameter changes may silently fail to apply.
3309 * @param params The bundle of parameters to set.
3310 * @throws IllegalStateException if in the Released state.
3312 public final void setParameters(@Nullable Bundle params) {
3313 if (params == null) {
3317 String[] keys = new String[params.size()];
3318 Object[] values = new Object[params.size()];
3321 for (final String key: params.keySet()) {
3323 values[i] = params.get(key);
3327 setParameters(keys, values);
3331 * Sets an asynchronous callback for actionable MediaCodec events.
3333 * If the client intends to use the component in asynchronous mode,
3334 * a valid callback should be provided before {@link #configure} is called.
3336 * When asynchronous callback is enabled, the client should not call
3337 * {@link #getInputBuffers}, {@link #getOutputBuffers},
3338 * {@link #dequeueInputBuffer(long)} or {@link #dequeueOutputBuffer(BufferInfo, long)}.
3340 * Also, {@link #flush} behaves differently in asynchronous mode. After calling
3341 * {@code flush}, you must call {@link #start} to "resume" receiving input buffers,
3342 * even if an input surface was created.
3344 * @param cb The callback that will run. Use {@code null} to clear a previously
3345 * set callback (before {@link #configure configure} is called and run
3346 * in synchronous mode).
3347 * @param handler Callbacks will happen on the handler's thread. If {@code null},
3348 * callbacks are done on the default thread (the caller's thread or the
3351 public void setCallback(@Nullable /* MediaCodec. */ Callback cb, @Nullable Handler handler) {
3353 synchronized (mListenerLock) {
3354 EventHandler newHandler = getEventHandlerOn(handler, mCallbackHandler);
3355 // NOTE: there are no callbacks on the handler at this time, but check anyways
3356 // even if we were to extend this to be callable dynamically, it must
3357 // be called when codec is flushed, so no messages are pending.
3358 if (newHandler != mCallbackHandler) {
3359 mCallbackHandler.removeMessages(EVENT_SET_CALLBACK);
3360 mCallbackHandler.removeMessages(EVENT_CALLBACK);
3361 mCallbackHandler = newHandler;
3364 } else if (mCallbackHandler != null) {
3365 mCallbackHandler.removeMessages(EVENT_SET_CALLBACK);
3366 mCallbackHandler.removeMessages(EVENT_CALLBACK);
3369 if (mCallbackHandler != null) {
3370 // set java callback on main handler
3371 Message msg = mCallbackHandler.obtainMessage(EVENT_SET_CALLBACK, 0, 0, cb);
3372 mCallbackHandler.sendMessage(msg);
3374 // set native handler here, don't post to handler because
3375 // it may cause the callback to be delayed and set in a wrong state.
3376 // Note that native codec may start sending events to the callback
3377 // handler after this returns.
3378 native_setCallback(cb);
3383 * Sets an asynchronous callback for actionable MediaCodec events on the default
3386 * Same as {@link #setCallback(Callback, Handler)} with handler set to null.
3387 * @param cb The callback that will run. Use {@code null} to clear a previously
3388 * set callback (before {@link #configure configure} is called and run
3389 * in synchronous mode).
3390 * @see #setCallback(Callback, Handler)
3392 public void setCallback(@Nullable /* MediaCodec. */ Callback cb) {
3393 setCallback(cb, null /* handler */);
3397 * Listener to be called when an output frame has rendered on the output surface
3399 * @see MediaCodec#setOnFrameRenderedListener
3401 public interface OnFrameRenderedListener {
3404 * Called when an output frame has rendered on the output surface.
3406 * <strong>Note:</strong> This callback is for informational purposes only: to get precise
3407 * render timing samples, and can be significantly delayed and batched. Some frames may have
3408 * been rendered even if there was no callback generated.
3410 * @param codec the MediaCodec instance
3411 * @param presentationTimeUs the presentation time (media time) of the frame rendered.
3412 * This is usually the same as specified in {@link #queueInputBuffer}; however,
3413 * some codecs may alter the media time by applying some time-based transformation,
3414 * such as frame rate conversion. In that case, presentation time corresponds
3415 * to the actual output frame rendered.
3416 * @param nanoTime The system time when the frame was rendered.
3418 * @see System#nanoTime
3420 public void onFrameRendered(
3421 @NonNull MediaCodec codec, long presentationTimeUs, long nanoTime);
3425 * Registers a callback to be invoked when an output frame is rendered on the output surface.
3427 * This method can be called in any codec state, but will only have an effect in the
3428 * Executing state for codecs that render buffers to the output surface.
3430 * <strong>Note:</strong> This callback is for informational purposes only: to get precise
3431 * render timing samples, and can be significantly delayed and batched. Some frames may have
3432 * been rendered even if there was no callback generated.
3434 * @param listener the callback that will be run
3435 * @param handler the callback will be run on the handler's thread. If {@code null},
3436 * the callback will be run on the default thread, which is the looper
3437 * from which the codec was created, or a new thread if there was none.
3439 public void setOnFrameRenderedListener(
3440 @Nullable OnFrameRenderedListener listener, @Nullable Handler handler) {
3441 synchronized (mListenerLock) {
3442 mOnFrameRenderedListener = listener;
3443 if (listener != null) {
3444 EventHandler newHandler = getEventHandlerOn(handler, mOnFrameRenderedHandler);
3445 if (newHandler != mOnFrameRenderedHandler) {
3446 mOnFrameRenderedHandler.removeMessages(EVENT_FRAME_RENDERED);
3448 mOnFrameRenderedHandler = newHandler;
3449 } else if (mOnFrameRenderedHandler != null) {
3450 mOnFrameRenderedHandler.removeMessages(EVENT_FRAME_RENDERED);
3452 native_enableOnFrameRenderedListener(listener != null);
3456 private native void native_enableOnFrameRenderedListener(boolean enable);
3458 private EventHandler getEventHandlerOn(
3459 @Nullable Handler handler, @NonNull EventHandler lastHandler) {
3460 if (handler == null) {
3461 return mEventHandler;
3463 Looper looper = handler.getLooper();
3464 if (lastHandler.getLooper() == looper) {
3467 return new EventHandler(this, looper);
3473 * MediaCodec callback interface. Used to notify the user asynchronously
3474 * of various MediaCodec events.
3476 public static abstract class Callback {
3478 * Called when an input buffer becomes available.
3480 * @param codec The MediaCodec object.
3481 * @param index The index of the available input buffer.
3483 public abstract void onInputBufferAvailable(@NonNull MediaCodec codec, int index);
3486 * Called when an output buffer becomes available.
3488 * @param codec The MediaCodec object.
3489 * @param index The index of the available output buffer.
3490 * @param info Info regarding the available output buffer {@link MediaCodec.BufferInfo}.
3492 public abstract void onOutputBufferAvailable(
3493 @NonNull MediaCodec codec, int index, @NonNull BufferInfo info);
3496 * Called when the MediaCodec encountered an error
3498 * @param codec The MediaCodec object.
3499 * @param e The {@link MediaCodec.CodecException} object describing the error.
3501 public abstract void onError(@NonNull MediaCodec codec, @NonNull CodecException e);
3504 * Called when the output format has changed
3506 * @param codec The MediaCodec object.
3507 * @param format The new output format.
3509 public abstract void onOutputFormatChanged(
3510 @NonNull MediaCodec codec, @NonNull MediaFormat format);
3513 private void postEventFromNative(
3514 int what, int arg1, int arg2, @Nullable Object obj) {
3515 synchronized (mListenerLock) {
3516 EventHandler handler = mEventHandler;
3517 if (what == EVENT_CALLBACK) {
3518 handler = mCallbackHandler;
3519 } else if (what == EVENT_FRAME_RENDERED) {
3520 handler = mOnFrameRenderedHandler;
3522 if (handler != null) {
3523 Message msg = handler.obtainMessage(what, arg1, arg2, obj);
3524 handler.sendMessage(msg);
3529 private native final void setParameters(@NonNull String[] keys, @NonNull Object[] values);
3532 * Get the codec info. If the codec was created by createDecoderByType
3533 * or createEncoderByType, what component is chosen is not known beforehand,
3534 * and thus the caller does not have the MediaCodecInfo.
3535 * @throws IllegalStateException if in the Released state.
3538 public MediaCodecInfo getCodecInfo() {
3539 // Get the codec name first. If the codec is already released,
3540 // IllegalStateException will be thrown here.
3541 String name = getName();
3542 synchronized (mCodecInfoLock) {
3543 if (mCodecInfo == null) {
3544 // Get the codec info for this codec itself first. Only initialize
3545 // the full codec list if this somehow fails because it can be slow.
3546 mCodecInfo = getOwnCodecInfo();
3547 if (mCodecInfo == null) {
3548 mCodecInfo = MediaCodecList.getInfoFor(name);
3556 private native final MediaCodecInfo getOwnCodecInfo();
3559 private native final ByteBuffer[] getBuffers(boolean input);
3562 private native final ByteBuffer getBuffer(boolean input, int index);
3565 private native final Image getImage(boolean input, int index);
3567 private static native final void native_init();
3569 private native final void native_setup(
3570 @NonNull String name, boolean nameIsType, boolean encoder);
3572 private native final void native_finalize();
3575 System.loadLibrary("media_jni");
3579 private long mNativeContext;
3582 public static class MediaImage extends Image {
3583 private final boolean mIsReadOnly;
3584 private final int mWidth;
3585 private final int mHeight;
3586 private final int mFormat;
3587 private long mTimestamp;
3588 private final Plane[] mPlanes;
3589 private final ByteBuffer mBuffer;
3590 private final ByteBuffer mInfo;
3591 private final int mXOffset;
3592 private final int mYOffset;
3594 private final static int TYPE_YUV = 1;
3596 private final int mTransform = 0; //Default no transform
3597 private final int mScalingMode = 0; //Default frozen scaling mode
3600 public int getFormat() {
3601 throwISEIfImageIsInvalid();
3606 public int getHeight() {
3607 throwISEIfImageIsInvalid();
3612 public int getWidth() {
3613 throwISEIfImageIsInvalid();
3618 public int getTransform() {
3619 throwISEIfImageIsInvalid();
3624 public int getScalingMode() {
3625 throwISEIfImageIsInvalid();
3626 return mScalingMode;
3630 public long getTimestamp() {
3631 throwISEIfImageIsInvalid();
3637 public Plane[] getPlanes() {
3638 throwISEIfImageIsInvalid();
3639 return Arrays.copyOf(mPlanes, mPlanes.length);
3643 public void close() {
3644 if (mIsImageValid) {
3645 java.nio.NioUtils.freeDirectBuffer(mBuffer);
3646 mIsImageValid = false;
3651 * Set the crop rectangle associated with this frame.
3653 * The crop rectangle specifies the region of valid pixels in the image,
3654 * using coordinates in the largest-resolution plane.
3657 public void setCropRect(@Nullable Rect cropRect) {
3659 throw new ReadOnlyBufferException();
3661 super.setCropRect(cropRect);
3666 @NonNull ByteBuffer buffer, @NonNull ByteBuffer info, boolean readOnly,
3667 long timestamp, int xOffset, int yOffset, @Nullable Rect cropRect) {
3668 mFormat = ImageFormat.YUV_420_888;
3669 mTimestamp = timestamp;
3670 mIsImageValid = true;
3671 mIsReadOnly = buffer.isReadOnly();
3672 mBuffer = buffer.duplicate();
3674 // save offsets and info
3679 // read media-info. See MediaImage2
3680 if (info.remaining() == 104) {
3681 int type = info.getInt();
3682 if (type != TYPE_YUV) {
3683 throw new UnsupportedOperationException("unsupported type: " + type);
3685 int numPlanes = info.getInt();
3686 if (numPlanes != 3) {
3687 throw new RuntimeException("unexpected number of planes: " + numPlanes);
3689 mWidth = info.getInt();
3690 mHeight = info.getInt();
3691 if (mWidth < 1 || mHeight < 1) {
3692 throw new UnsupportedOperationException(
3693 "unsupported size: " + mWidth + "x" + mHeight);
3695 int bitDepth = info.getInt();
3696 if (bitDepth != 8) {
3697 throw new UnsupportedOperationException("unsupported bit depth: " + bitDepth);
3699 int bitDepthAllocated = info.getInt();
3700 if (bitDepthAllocated != 8) {
3701 throw new UnsupportedOperationException(
3702 "unsupported allocated bit depth: " + bitDepthAllocated);
3704 mPlanes = new MediaPlane[numPlanes];
3705 for (int ix = 0; ix < numPlanes; ix++) {
3706 int planeOffset = info.getInt();
3707 int colInc = info.getInt();
3708 int rowInc = info.getInt();
3709 int horiz = info.getInt();
3710 int vert = info.getInt();
3711 if (horiz != vert || horiz != (ix == 0 ? 1 : 2)) {
3712 throw new UnsupportedOperationException("unexpected subsampling: "
3713 + horiz + "x" + vert + " on plane " + ix);
3715 if (colInc < 1 || rowInc < 1) {
3716 throw new UnsupportedOperationException("unexpected strides: "
3717 + colInc + " pixel, " + rowInc + " row on plane " + ix);
3721 buffer.position(mBuffer.position() + planeOffset
3722 + (xOffset / horiz) * colInc + (yOffset / vert) * rowInc);
3723 buffer.limit(buffer.position() + Utils.divUp(bitDepth, 8)
3724 + (mHeight / vert - 1) * rowInc + (mWidth / horiz - 1) * colInc);
3725 mPlanes[ix] = new MediaPlane(buffer.slice(), rowInc, colInc);
3728 throw new UnsupportedOperationException(
3729 "unsupported info length: " + info.remaining());
3732 if (cropRect == null) {
3733 cropRect = new Rect(0, 0, mWidth, mHeight);
3735 cropRect.offset(-xOffset, -yOffset);
3736 super.setCropRect(cropRect);
3739 private class MediaPlane extends Plane {
3740 public MediaPlane(@NonNull ByteBuffer buffer, int rowInc, int colInc) {
3747 public int getRowStride() {
3748 throwISEIfImageIsInvalid();
3753 public int getPixelStride() {
3754 throwISEIfImageIsInvalid();
3760 public ByteBuffer getBuffer() {
3761 throwISEIfImageIsInvalid();
3765 private final int mRowInc;
3766 private final int mColInc;
3767 private final ByteBuffer mData;
3771 public final static class MetricsConstants
3773 private MetricsConstants() {}
3776 * Key to extract the codec being used
3777 * from the {@link MediaCodec#getMetrics} return value.
3778 * The value is a String.
3780 public static final String CODEC = "android.media.mediacodec.codec";
3783 * Key to extract the MIME type
3784 * from the {@link MediaCodec#getMetrics} return value.
3785 * The value is a String.
3787 public static final String MIME_TYPE = "android.media.mediacodec.mime";
3790 * Key to extract what the codec mode
3791 * from the {@link MediaCodec#getMetrics} return value.
3792 * The value is a String. Values will be one of the constants
3793 * {@link #MODE_AUDIO} or {@link #MODE_VIDEO}.
3795 public static final String MODE = "android.media.mediacodec.mode";
3798 * The value returned for the key {@link #MODE} when the
3799 * codec is a audio codec.
3801 public static final String MODE_AUDIO = "audio";
3804 * The value returned for the key {@link #MODE} when the
3805 * codec is a video codec.
3807 public static final String MODE_VIDEO = "video";
3810 * Key to extract the flag indicating whether the codec is running
3811 * as an encoder or decoder from the {@link MediaCodec#getMetrics} return value.
3812 * The value is an integer.
3813 * A 0 indicates decoder; 1 indicates encoder.
3815 public static final String ENCODER = "android.media.mediacodec.encoder";
3818 * Key to extract the flag indicating whether the codec is running
3819 * in secure (DRM) mode from the {@link MediaCodec#getMetrics} return value.
3820 * The value is an integer.
3822 public static final String SECURE = "android.media.mediacodec.secure";
3825 * Key to extract the width (in pixels) of the video track
3826 * from the {@link MediaCodec#getMetrics} return value.
3827 * The value is an integer.
3829 public static final String WIDTH = "android.media.mediacodec.width";
3832 * Key to extract the height (in pixels) of the video track
3833 * from the {@link MediaCodec#getMetrics} return value.
3834 * The value is an integer.
3836 public static final String HEIGHT = "android.media.mediacodec.height";
3839 * Key to extract the rotation (in degrees) to properly orient the video
3840 * from the {@link MediaCodec#getMetrics} return.
3841 * The value is a integer.
3843 public static final String ROTATION = "android.media.mediacodec.rotation";
OSDN Git Service
