--- /dev/null
+page.title=Testing Display Performance
+
+@jd:body
+
+
+<div id="qv-wrapper">
+ <div id="qv">
+ <h2>In this document</h2>
+ <ol>
+ <li><a href="#measure">Measuring UI Performance</a>
+ <ul>
+ <li><a href="#aggregate">Aggregate frame stats</a></li>
+ <li><a href="#timing-info">Precise frame timing info</a></li>
+ <li><a href="#timing-dump">Simple frame timing dump</a></li>
+ <li><a href="#collection-window">Controlling the window of stat collection</a></li>
+ <li><a href="#diagnose">Diagnosing performance regressions</a></li>
+ <li><a href="#resources">Additional resources</a></li>
+ </ul>
+ </li>
+ <li><a href="#automate">Automating UI Perfomance Tests</a>
+ <ul>
+ <li><a href="#ui-tests">Setting up UI tests</a></li>
+ <li><a href="#automated-tests">Setting up automated UI testing</a></li>
+ <li><a href="#triage">Triaging and fixing observed problems</a></li>
+ </ul>
+ </li>
+ </ol>
+ </div>
+</div>
+
+
+<p>
+ User interface (UI) performance testing ensures that your app not only meets its functional
+ requirements, but that user interactions with your app are buttery smooth, running at a
+ consistent 60 frames per second (<a href=
+ "https://www.youtube.com/watch?v=CaMTIgxCSqU&index=25&list=PLWz5rJ2EKKc9CBxr3BVjPTPoDPLdPIFCE">why
+ 60fps?</a>), without any dropped or delayed frames, or as we like to call it, <em>jank</em>. This
+ document explains tools available to measure UI performance, and lays out an approach to
+ integrate UI performance measurements into your testing practices.
+</p>
+
+
+<h2 id="measure">Measuring UI Performance</h2>
+
+<p>
+ In order to improve performance you first need the ability to measure the performance of
+ your system, and then diagnose and identify problems that may arrive from various parts of your
+ pipeline.
+</p>
+
+<p>
+ <em><a href="https://source.android.com/devices/tech/debug/dumpsys.html">dumpsys</a></em> is an
+ Android tool that runs on the device and dumps interesting information about the status of system
+ services. Passing the <em>gfxinfo</em> command to dumpsys provides an output in logcat with
+ performance information relating to frames of animation that are occurring during the recording
+ phase.
+</p>
+
+<pre>
+> adb shell dumpsys gfxinfo <PACKAGE_NAME>
+</pre>
+
+<p>
+ This command can produce multiple different variants of frame timing data.
+</p>
+
+<h3 id="aggregate">Aggregate frame stats</h3>
+
+<p>
+ With the M Preview the command prints out aggregated analysis of frame data to logcat, collected
+ across the entire lifetime of the process. For example:
+</p>
+
+<pre class="noprettyprint">
+Stats since: 752958278148ns
+Total frames rendered: 82189
+Janky frames: 35335 (42.99%)
+90th percentile: 34ms
+95th percentile: 42ms
+99th percentile: 69ms
+Number Missed Vsync: 4706
+Number High input latency: 142
+Number Slow UI thread: 17270
+Number Slow bitmap uploads: 1542
+Number Slow draw: 23342
+</pre>
+
+<p>
+ These high level statistics convey at a high level the rendering performance of the app, as well
+ as its stability across many frames.
+</p>
+
+
+<h3 id="timing-info">Precise frame timing info</h3>
+
+<p>
+ With the M Preview comes a new command for gfxinfo, and that’s <em>framestats</em> which provides
+ extremely detailed frame timing information from recent frames, so that you can track down and
+ debug problems more accurately.
+</p>
+
+<pre>
+>adb shell dumpsys gfxinfo <PACKAGE_NAME> framestats
+</pre>
+
+<p>
+ This command prints out frame timing information, with nanosecond timestamps, from the last 120
+ frames produced by the app. Below is example raw output from adb dumpsys gfxinfo
+ <PACKAGE_NAME> framestats:
+</p>
+
+<pre class="noprettyprint">
+0,49762224585003,49762241251670,9223372036854775807,0,49762257627204,49762257646058,49762257969704,49762258002100,49762265541631,49762273951162,49762300914808,49762303675954,
+0,49762445152142,49762445152142,9223372036854775807,0,49762446678818,49762446705589,49762447268818,49762447388037,49762453551527,49762457134131,49762474889027,49762476150120,
+0,49762462118845,49762462118845,9223372036854775807,0,49762462595381,49762462619287,49762462919964,49762462968454,49762476194547,49762476483454,49762480214964,49762480911527,
+0,49762479085548,49762479085548,9223372036854775807,0,49762480066370,49762480099339,49762481013089,49762481085850,49762482232152,49762482478350,49762485657620,49762486116683,
+</pre>
+
+<p>
+ Each line of this output represents a frame produced by the app. Each line has a fixed number of
+ columns describing time spent in each stage of the frame-producing pipeline. The next section
+ describes this format in detail, including what each column represents.
+</p>
+
+
+<h4 id="fs-data-format">Framestats data format</h4>
+
+<p>
+ Since the block of data is output in CSV format, it's very straightforward to paste it to your
+ spreadsheet tool of choice, or collect and parse with a script. The following table explains the
+ format of the output data columns. All timestamps are in nanoseconds.
+</p>
+
+<ul>
+ <li>FLAGS
+ <ul>
+ <li>Rows with a ‘0’ for the FLAGS column can have their total frame time computed by
+ subtracting the INTENDED_VSYNC column from the FRAME_COMPLETED column.
+ </li>
+
+ <li>If this is non-zero the row should be ignored, as the frame has been determined as being
+ an outlier from normal performance, where it is expected that layout & draw take longer
+ than 16ms. Here are a few reasons this could occur:
+ <ul>
+ <li>The window layout changed (such as the first frame of the application or after a
+ rotation)
+ </li>
+
+ <li>It is also possible the frame was skipped in which case some of the values will have
+ garbage timestamps. A frame can be skipped if for example it is out-running 60fps or if
+ nothing on-screen ended up being dirty, this is not necessarily a sign of a problem in
+ the app.
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+
+ <li>VSYNC
+ <ul>
+ <li>The time value that was used in all the vsync listeners and drawing for the frame
+ (Choreographer frame callbacks, animations, View.getDrawingTime(), etc…)
+ </li>
+
+ <li>To understand more about VSYNC and how it influences your application, check out the
+ <a href=
+ "https://www.youtube.com/watch?v=1iaHxmfZGGc&list=PLOU2XLYxmsIKEOXh5TwZEv89aofHzNCiu&index=23">
+ Understanding VSYNC</a> video.
+ </li>
+ </ul>
+ </li>
+
+
+ <li>INTENDED_VSYNC
+ <ul>
+ <li>The intended start point for the frame. If this value is different from VSYNC, there
+ was work occurring on the UI thread that prevented it from responding to the vsync signal
+ in a timely fashion.
+ </li>
+ </ul>
+ </li>
+
+ <li>OLDEST_INPUT_EVENT
+ <ul>
+ <li>The timestamp of the oldest input event in the input queue, or Long.MAX_VALUE if
+ there were no input events for the frame.
+ </li>
+
+ <li>This value is primarily intended for platform work and has limited usefulness to app
+ developers.
+ </li>
+ </ul>
+ </li>
+
+ <li>NEWEST_INPUT_EVENT
+ <ul>
+ <li>The timestamp of the newest input event in the input queue, or 0 if there were no
+ input events for the frame.
+ </li>
+
+ <li>This value is primarily intended for platform work and has limited usefulness to app
+ developers.
+ </li>
+
+ <li>However it’s possible to get a rough idea of how much latency the app is adding by
+ looking at (FRAME_COMPLETED - NEWEST_INPUT_EVENT).
+ </li>
+ </ul>
+ </li>
+
+ <li>HANDLE_INPUT_START
+ <ul>
+ <li>The timestamp at which input events were dispatched to the application.
+ </li>
+
+ <li>By looking at the time between this and ANIMATION_START it is possible to measure how
+ long the application spent handling input events.
+ </li>
+
+ <li>If this number is high (>2ms), this indicates the app is spending an unusually
+ long time processing input events, such as View.onTouchEvent(), which may indicate this
+ work needs to be optimized, or offloaded to a different thread. Note that there are some
+ scenarios, such as click events that launch new activities or similar, where it is
+ expected and acceptable that this number is large.
+ </li>
+ </ul>
+ </li>
+
+ <li>ANIMATION_START
+ <ul>
+ <li>The timestamp at which animations registered with Choreographer were run.
+ </li>
+
+ <li>By looking at the time between this and PERFORM_TRANVERSALS_START it is possible to
+ determine how long it took to evaluate all the animators (ObjectAnimator,
+ ViewPropertyAnimator, and Transitions being the common ones) that are running.
+ </li>
+
+ <li>If this number is high (>2ms), check to see if your app has written any custom
+ animators or what fields ObjectAnimators are animating and ensure they are appropriate
+ for an animation.
+ </li>
+
+ <li>To learn more about Choreographer, check out the <a href=
+ "https://developers.google.com/events/io/sessions/325418001">For Butter or Worse</a>
+ video.
+ </li>
+ </ul>
+ </li>
+
+ <li>PERFORM_TRAVERSALS_START
+ <ul>
+ <li>If you subtract out DRAW_START from this value, you can extract how long the layout
+ & measure phases took to complete. (note, during a scroll, or animation, you would
+ hope this should be close to zero..)
+ </li>
+
+ <li>To learn more about the measure & layout phases of the rendering pipeline, check
+ out the <a href=
+ "https://www.youtube.com/watch?v=we6poP0kw6E&list=PLOU2XLYxmsIKEOXh5TwZEv89aofHzNCiu&index=27">
+ Invalidations, Layouts and Performance</a> video
+ </li>
+ </ul>
+ </li>
+
+ <li>DRAW_START
+ <ul>
+ <li>The time at which the draw phase of performTraversals started. This is the start
+ point of recording the display lists of any views that were invalidated.
+ </li>
+
+ <li>The time between this and SYNC_START is how long it took to call View.draw() on all
+ the invalidated views in the tree.
+ </li>
+
+ <li>For more information on the drawing model, see <a href=
+ "{@docRoot}guide/topics/graphics/hardware-accel.html#hardware-model">Hardware Acceleration</a>
+ or the <a href=
+ "https://www.youtube.com/watch?v=we6poP0kw6E&list=PLOU2XLYxmsIKEOXh5TwZEv89aofHzNCiu&index=27">
+ Invalidations, Layouts and Performance</a> video
+ </li>
+ </ul>
+ </li>
+
+ <li>SYNC_START
+ <ul>
+ <li>The time at which the sync phase of the drawing started.
+ </li>
+
+ <li>If the time between this and ISSUE_DRAW_COMMANDS_START is substantial (>0.4ms or
+ so), it typically indicates a lot of new Bitmaps were drawn which must be uploaded to the
+ GPU.
+ </li>
+
+ <li>To understand more about the sync phase, check out the <a href=
+ "https://www.youtube.com/watch?v=VzYkVL1n4M8&index=24&list=PLOU2XLYxmsIKEOXh5TwZEv89aofHzNCiu">
+ Profile GPU Rendering</a> video
+ </li>
+ </ul>
+ </li>
+
+ <li>ISSUE_DRAW_COMMANDS_START
+ <ul>
+ <li>The time at which the hardware renderer started issuing drawing commands to the GPU.
+ </li>
+
+ <li>The time between this and FRAME_COMPLETED gives a rough idea of how much GPU work the
+ app is producing. Problems like too much overdraw or inefficient rendering effects show
+ up here.
+ </li>
+ </ul>
+ </li>
+
+ <li>SWAP_BUFFERS
+ <ul>
+ <li>The time at which eglSwapBuffers was called, relatively uninteresting outside of
+ platform work.
+ </li>
+ </ul>
+ </li>
+
+ <li>FRAME_COMPLETED
+ <ul>
+ <li>All done! The total time spent working on this frame can be computed by doing
+ FRAME_COMPLETED - INTENDED_VSYNC.
+ </li>
+ </ul>
+ </li>
+
+</ul>
+
+<p>
+ You can use this data in different ways. One simple but useful visualization is a
+ histogram showing the distribution of frames times (FRAME_COMPLETED - INTENDED_VSYNC) in
+ different latency buckets, see figure below. This graph tells us at a glance that most
+ frames were very good - well below the 16ms deadline (depicted in red), but a few frames
+ were significantly over the deadline. We can look at changes in this histogram over time
+ to see wholesale shifts or new outliers being created. You can also graph input latency,
+ time spent in layout, or other similar interesting metrics based on the many timestamps
+ in the data.
+</p>
+
+<img src="{@docRoot}preview/images/perf-test-framestats.png">
+
+
+<h3 id="timing-dump">Simple frame timing dump</h3>
+
+<p>
+ If <strong>Profile GPU rendering</strong> is set to <strong>In adb shell dumpsys gfxinfo</strong>
+ in Developer Options, the <code>adb shell dumpsys gfxinfo</code> command prints out timing
+ information for the most recent 120 frames, broken into a few different categories with
+ tab-separated-values. This data can be useful for indicating which parts of the drawing pipeline
+ may be slow at a high level.
+</p>
+
+<p>
+ Similar to <a href="#fs-data-format">framestats</a> above, it's very
+ straightforward to paste it to your spreadsheet tool of choice, or collect and parse with
+ a script. The following graph shows a breakdown of where many frames produced by the app
+ were spending their time.
+</p>
+
+<img src="{@docRoot}preview/images/perf-test-frame-latency.png">
+
+<p>
+ The result of running gfxinfo, copying the output, pasting it into a spreadsheet
+ application, and graphing the data as stacked bars.
+</p>
+
+<p>
+ Each vertical bar represents one frame of animation; its height represents the number of
+ milliseconds it took to compute that frame of animation. Each colored segment of the bar
+ represents a different stage of the rendering pipeline, so that you can see what parts of
+ your application may be creating a bottleneck. For more information on understanding the
+ rendering pipeline, and how to optimize for it, see the <a href=
+ "https://www.youtube.com/watch?v=we6poP0kw6E&index=27&list=PLWz5rJ2EKKc9CBxr3BVjPTPoDPLdPIFCE">
+ Invalidations Layouts and Performance</a> video.
+</p>
+
+
+<h3 id="collection-window">Controlling the window of stat collection</h3>
+
+<p>
+ Both the framestats and simple frame timings gather data over a very short window - about
+ two seconds worth of rendering. In order to precisely control this window of time - for
+ example, to constrain the data to a particular animation - you can reset all counters,
+ and aggregate statistics gathered.
+</p>
+
+<pre>
+>adb shell dumpsys gfxinfo <PACKAGE_NAME> reset
+</pre>
+
+<p>
+ This can also be used in conjunction with the dumping commands themselves to collect and
+ reset at a regular cadence, capturing less-than-two-second windows of frames
+ continuously.
+</p>
+
+
+<h3 id="diagnose">Diagnosing performance regressions</h3>
+
+<p>
+ Identification of regressions is a good first step to tracking down problems, and
+ maintaining high application health. However, dumpsys just identifies the existence and
+ relative severity of problems. You still need to diagnose the particular cause of the
+ performance problems, and find appropriate ways to fix them. For that, it’s highly
+ recommended to use the <a href="{@docRoot}tools/help/systrace.html">systrace</a> tool.
+</p>
+
+
+<h3 id="resources">Additional resources</h3>
+
+<p>
+ For more information on how Android’s rendering pipeline works, common problems that you
+ can find there, and how to fix them, some of the following resources may be useful to
+ you:
+</p>
+
+<ul>
+ <li>Rendering Performance 101
+ </li>
+ <li>Why 60fps?
+ </li>
+ <li>Android UI and the GPU
+ </li>
+ <li>Invalidations Layouts and performance
+ </li>
+ <li>Analyzing UI Performance with Systrace
+ </li>
+</ul>
+
+
+<h2 id="automate">Automating UI Perfomance Tests</h2>
+
+<p>
+ One approach to UI Performance testing is to simply have a human tester perform a set of
+ user operations on the target app, and either visually look for jank, or spend an very
+ large amount of time using a tool-driven approach to find it. But this manual approach is
+ fraught with peril - human ability to perceive frame rate changes varies tremendously,
+ and this is also time consuming, tedious, and error prone.
+</p>
+
+<p>
+ A more efficient approach is to log and analyze key performance metrics from automated UI
+ tests. The Android M developer preview includes new logging capabilities which make it
+ easy to determine the amount and severity of jank in your application’s animations, and
+ that can be used to build a rigorous process to determine your current performance and
+ track future performance objectives.
+</p>
+
+<p>
+ This article walks you through a recommended approach to using that data to automate your
+ performance testing.
+</p>
+
+<p>
+ This is mostly broken down into two key actions. Firstly, identifying what you're
+ testing, and how you’re testing it. and Secondly, setting up, and maintaining an
+ automated testing environment.
+</p>
+
+
+<h3 id="ui-tests">Setting up UI tests</h3>
+
+<p>
+ Before you can get started with automated testing, it’s important to determine a few high
+ level decisions, in order to properly understand your test space, and needs you may have.
+</p>
+
+<h4>
+ Identify key animations / flows to test
+</h4>
+
+<p>
+ Remember that bad performance is most visible to users when it interrupts a smooth
+ animation. As such, when identifying what types of UI actions to test for, it’s useful to
+ focus on the key animations that users see most, or are most important to their
+ experience. For example, here are some common scenarios that may be useful to identify:
+</p>
+
+<ul>
+ <li>Scrolling a primary ListView or RecyclerView
+ </li>
+
+ <li>Animations during async wait cycles
+ </li>
+
+ <li>Any animation that may have bitmap loading / manipulation in it
+ </li>
+
+ <li>Animations including Alpha Blending
+ </li>
+
+ <li>Custom View drawing with Canvas
+ </li>
+</ul>
+
+<p>
+ Work with engineers, designers, and product managers on your team to prioritize these key
+ product animations for test coverage.
+</p>
+
+<h4>
+ Define your future objectives and track against them
+</h4>
+
+<p>
+ From a high-level, it may be critical to identify your specific performance goals, and
+ focus on writing tests, and collecting data around them. For example:
+</p>
+
+<ul>
+ <li>Do you just want to begin tracking UI performance for the first time to learn more?
+ </li>
+
+ <li>Do you want to prevent regressions that might be introduced in the future?
+ </li>
+
+ <li>Are you at 90% of smooth frames today and want to get to 98% this quarter?
+ </li>
+
+ <li>Are you at 98% smooth frames and don’t want to regress?
+ </li>
+
+ <li>Is your goal to improve performance on low end devices?
+ </li>
+</ul>
+
+<p>
+ In all of these cases, you’ll want historical tracking which shows performance across
+ multiple versions of your application.
+</p>
+
+<h4>
+ Identify devices to test on
+</h4>
+
+<p>
+ Application performance varies depending on the device it's running on. Some devices may
+ contain less memory, less powerful GPUs, or slower CPU chips. This means that animations
+ which may perform well on one set of hardware, may not on others, and worse, may be a
+ result of a bottleneck in a different part of the pipeline. So, to account for this
+ variation in what a user might see, pick a range of devices to execute tests on, both
+ current high end devices, low end devices, tablets, etc. Look for variation in CPU
+ performance, RAM, screen density, size, and so on. Tests that pass on a high end device
+ may fail on a low end device.
+</p>
+
+<h4>
+ Basic frameworks for UI Testing
+</h4>
+
+<p>
+ Tool suites like <a href=
+ "https://developer.android.com/tools/testing-support-library/index.html">UIAutomator</a>,
+ and <a href="https://code.google.com/p/android-test-kit/">Espresso</a> are built to help
+ automate the action of a user moving through your application. These are simple
+ frameworks which mimic user interaction with your device. To use these frameworks, you
+ effectively create unique scripts, which run through a set of user-actions, and play them
+ out on the device itself.
+</p>
+
+<p>
+ By combining these automated tests, alongside <code>dumpsys gfxinfo</code> you can quickly
+ create a reproducible system that allows you to execute a test, and measure the
+ performance information of that particular condition.
+</p>
+
+
+<h3 id="automated-tests">Setting up automated UI testing</h3>
+
+<p>
+ Once you have the ability to execute a UI test, and a pipeline to gather the data from a
+ single test, the next important step is to embrace a framework which can execute that
+ test multiple times, across multiple devices, and aggregate the resulting performance
+ data for further analysis by your development team.
+</p>
+
+<h4>
+ A framework for test automation
+</h4>
+
+<p>
+ It’s worth noting that UI testing frameworks (like <a href=
+ "https://developer.android.com/tools/testing-support-library/index.html">UIAutomator</a>)
+ run on the target device/emulator directly. While performance gathering information done
+ by <em>dumpsys gfxinfo</em> is driven by a host machine, sending commands over ADB. To
+ help bridge the automation of these separate entities, <a href=
+ "{@docRoot}tools/help/monkeyrunner_concepts.html">MonkeyRunner</a> framework was
+ developed; A scripting system that runs on your host machine, which can issue commands to
+ a set of connected devices, as well as receive data from them.
+</p>
+
+<p>
+ Building a set of scripts for proper Automation of UI Performance testing, at a minimum,
+ should be able to utilize monkeyRunner to accomplish the following tasks:
+</p>
+
+<ul>
+ <li>Load & Launch a desired APK to a target device, devices, or emulator.
+ </li>
+
+ <li>Launch a UIAutomator UI test, and allow it to be executed
+ </li>
+
+ <li>Collect performance information through <em>dumpsys gfxinfo</em><em>.</em>
+ </li>
+
+ <li>Aggregate information and display it back in a useful fashion to the developer.
+ </li>
+</ul>
+
+
+<h3 id="triage">Triaging and fixing observed problems</h3>
+
+<p>
+ Once problem patterns or regressions are identified, the next step is identifying and
+ applying the fix. If your automated test framework preserves precise timing breakdowns
+ for frames, it can help you scrutinize recent suspicious code/layout changes (in the case
+ of regression), or narrow down the part of the system you’re analyzing when you switch to
+ manual investigation. For manual investigation, <a href=
+ "{@docRoot}tools/help/systrace.html">systrace</a> is a great place to start, showing
+ precise timing information about every stage of the rendering pipeline, every thread and
+ core in the system, as well as any custom event markers you define.
+</p>
+
+<h4>
+ Properly profiling temporal timings
+</h4>
+
+<p>
+ It is important to note the difficulties in obtaining and measuring timings that come from
+ rendering performance. These numbers are, by nature, non deterministic, and often
+ fluctuate depending on the state of the system, amount of memory available, thermal
+ throttling, and the last time a sun flare hit your area of the earth. The point is that
+ you can run the same test, twice and get slightly different numbers that may be close to
+ each other, but not exact.
+</p>
+
+<p>
+ Properly gathering and profiling data in this manner means running the same test,
+ multiple times, and accumulating the results as an average, or median value. (for the
+ sake of simplicity, let’s call this a ‘batch’) This gives you the rough approximation of
+ the performance of the test, while not needing exact timings.
+</p>
+
+<p>
+ Batches can be used between code changes to see the relative impact of those changes on
+ performance. If the average frame rate for the pre-change Batch is larger than the
+ post-change batch, then you generally have an overall win wrt performance for that
+ particular change.
+</p>
+
+<p>
+ This means that any Automated UI testing you do should take this concept into
+ consideration, and also account for any anomalies that might occur during a test. For
+ example, if your application performance suddenly dips, due to some device issue (that
+ isn’t caused by your application) then you may want to re-run the batch in order to get
+ less chaotic timings.
+</p>
+
+<p>
+ So, how many times should you run a test, before the measurements become meaningful? 10
+ times should be the minimum, with higher numbers like 50 or 100 yielding more accurate
+ results (of course, you’re now trading off time for accuracy)
+</p>
OSDN Git Service
