OSDN Git Service

roomservice delivers you lunch combos from the CyanogenMod github.
authorKoushik Dutta <koushd@gmail.com>
Sun, 27 Nov 2011 02:51:42 +0000 (18:51 -0800)
committerSteve Kondik <steve@cyngn.com>
Sun, 4 Sep 2016 04:47:36 +0000 (21:47 -0700)
fix roomservice formatting

support product names with _ in them

fix roomservice to handle pagination

Change-Id: I4923c2f768094dbad8e06a72d9f27d46414030ab

roomservice: Add lightweight dependencies to repositories

Roomservice can already fetch your cm_<device> without the need for a
manifest entry.

However, when working with common repositories, there is no way of
actually fetching them without adding to the manifest. This patch
introduces a lightweight dependency system. Each repository can have a
cm.dependencies in the following json format:

[
  {
    "repository": "repository_name_on_cm_organization"
    "target_path": "target/path"
  },
  ...
]

For instance, for cm_anzu I need android_device_semc_msm7x30-common and
android_device_semc_mogami-common. I would add both to cm.dependencies
as follows:

[
  {
    "repository": "android_device_semc_msm7x30-common",
    "target_path": "device/semc/msm7x30-common"
  },
  {
    "repository": "android_device_semc_mogami-common",
    "target_path": "device/semc/mogami-common"
  }
]

Roomservice would then fetch the anzu repository, parse the dependency
files and add/fetch/sync these additional repositories if they don't
exist already.

This also adds pretty printing to the output xml.

Change-Id: I9cc847adfc717a06439bc6094213ed6492343158

roomservice: Add branch support to cm.dependencies

Allow the cm.dependencies entries to provide an optional "branch" for
the repository dependencies. Added to fully support
http://wiki.cyanogenmod.com/wiki/Integrated_kernel_building

Change-Id: I35b51920d296afa329411af6172c7bd9aeef4af8

roomservice: Fill in dependencies for already-deployed repositories

Change-Id: I01fd408c9c4bfa78097c7f848b2556d2b2b180f3

roomservice: Extend dependency-checks to devices in main manifest

CM currently keeps devices inherited from AOSP in the main manifest,
so take that into account as well when checking device paths

Change-Id: I9663f283617f237428b4eaa0cd60b5de2b86a7b9

make compatible with github v3 api

Change-Id: Iff6f1f9099cdc5d2b49e04000b5fe3d04aa5d7e4

Fixed build for full-eng

Previously
Traceback (most recent call last):
  File "build/tools/roomservice.py", line 153, in <module>
    repo_path = get_from_manifest(device)
NameError: name 'device' is not defined

** Don't have a product spec for: 'full'
** Do you have the right repo manifest?

Now
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=4.0.4
TARGET_PRODUCT=full
TARGET_BUILD_VARIANT=eng
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a
HOST_ARCH=x86
HOST_OS=linux
HOST_BUILD_TYPE=release
BUILD_ID=IMM76L
============================================

Change-Id: Ib513705aba9a7a52a971ab64102ecbe9fddfb97a

roomservice: Bump github request per_page to 100

Change the number of repos per page from the default 30
to 100.

We seem to be hitting the rate limit on the jenkins server.

Change-Id: Ie733feaa0414cbfebb7efcfc1e24d94e1e466d1b

roomservice: Add support for netrc

Change-Id: I1f5e11e40125abd0c4e4c8d8294d4fc09bfdc30a

roomservice: Handle missing netrc file

Change-Id: If981fe79dc3e2191434301239b0cd585be8b4730

roomservice.py: Verbose error when a branch is missing.
Also add ROOMSERVICE_BRANCHES environment variable to use fallback branches.

Change-Id: I3c2b1d79fc185c1f1e1d658e5ca4f78e688780e2

roomservice.py: Fixups around fallback branches not being used by dependencies when ROOMSERVICE_BRANCHES is defined.

Change-Id: Ifb42a023cae5f62ac8f9cf7832125b91b431169c

roomservice: Allow following up tag references

This is now needed for release builds

Change-Id: I8c5f87341059b3b15ee853312b71df73790ad0d8

build: local_manifest.xml deprecated, use local_manifests

Patch Set 2:- Use roomservice.xml instead of cm.xml.

Change-Id: I3d8a6ef3907b92808662cbba912cea5ed38d0bde

Fix fallback branch search in roomservice

If you provided a fallback branch to roomservice via the
ROOMSERVICE_BRANCHES environment var the branch search would fail
if the device repo had any tags.

Fixed this by appending the tag search results to the branch
search results instead of overwriting them

Change-Id: I73a11af1500bd04e346f08ec3f83454502f3a169

roomservice: Fix wrong assumption about path of active manifest

Change-Id: Id740ff4b848e6ccbfd658be4846197b8ca519237

roomservice: When validating the presence of a repo, check main manifest too

Change-Id: If680536484074b473458723d93e783d074d7f669

roomservice: Bump devices per page to 200

Limit was reached again when attempting to
lunch various projects, such as steelhead.

Change-Id: I2f3b9705e07e1e47b86857aeb383cf7c99fcdbdc
Signed-off-by: William Roberts <bill.c.roberts@gmail.com>
roomservice: Fix assumptions about the branch naming

We can't just split from the last slash anymore, since we're using
those to distinguish the stabilization branches

Change-Id: Ia175dd317f508e99b275b56e9c83bd4729a75ddb

roomservice: Add recursive dependencies

Dependency repositories can now have dependencies themselves

Change-Id: I33a28709170da52bc98f4a62387927e3a11b2450

roomservice: python3 support

Change-Id: I7621818ba7ed997676728fe865f37a25b3a5b8b5
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
roomservice: Improve new device retrieval

Without credentials, GitHub's search API limits requests to 60/hr.
The existing method to add a new device is to fetch JSON-formatted info
for ALL CM repositories and then search for the device. In doing so,
more than 10 pages of results are returned (i.e. more than 10 requests
per device). This is clumsy, slow, and limits use of roomservice to
only ~5 devices per hour.

Instead, only return search results for repositories that have the
device name in the repository name. Then, one device = one request.
It's faster and allows closer to 60 device setups / hr.

Additional bailouts are included to stop the script earlier than later
if a device is not found.

Change-Id: I7f914d7ede82da0f100d9fd6cf8b603177962e48

roomservice: Fix search of devices forked to CM

Commit "roomservice: Improve new device retrieval" introduced a
regression where repositories that were forked to CM were omitted from
search results. This fixes that issue.

Change-Id: I7bf54129b5da1749abe5b2b9a492cb93e6ee41a6

roomservice: Improve error handling on search

Use the relevant forms of 'except' for urllib and parsing instead of a
general except which was performed too late anyways.

Change-Id: Ia1fc89dd5a8a703fc0175aef7b6dd013a44a2c8e

roomservice: use force-sync when adding projects with roomservice

This works around the error
GitError: --force-sync not enabled; cannot overwrite a local work tree
when using roomservice.

Since this should only trigger when the device repos haven't been
checked out yet, we can use force-sync to overwrite any roomservice
device paths in the .repo directory.

Change-Id: Iac54a8a2f2913f82f8ca6497b8785a9d5769640b
Ticket: CYNGNOS-735

roomservice: check uniqueness by path, not name

For repos such as hardware/qcom/media-caf we are using the same name
with different branches for different paths.

for eg.
  CyanogenMod/hardware_qcom_media-caf(branch:8994)
   - fetch to : /hardware/qcom/media-caf/8994
  CyanogenMod/hardware_qcom_media-caf(branch:8960)
   - fetch to : /hardware/qcom/media-caf/8960

For such cases roomservice won't pick up a new path
if one already exists.

We should check for unique by target path instead.

Change-Id: I89e561ca9a2d57ede8cf782f431a8e829ea47ee5
Signed-off-by: Arnav Gupta <championswimmer@gmail.com>
envsetup.sh
tools/roomservice.py [new file with mode: 0755]

index 9ddf869..ff7312d 100644 (file)
@@ -606,6 +606,17 @@ function lunch()
     build_build_var_cache
     if [ $? -ne 0 ]
     then
+        # if we can't find a product, try to grab it off the CM github
+        T=$(gettop)
+        pushd $T > /dev/null
+        build/tools/roomservice.py $product
+        popd > /dev/null
+        check_product $product
+    else
+        build/tools/roomservice.py $product true
+    fi
+    if [ $? -ne 0 ]
+    then
         echo
         echo "** Don't have a product spec for: '$product'"
         echo "** Do you have the right repo manifest?"
diff --git a/tools/roomservice.py b/tools/roomservice.py
new file mode 100755 (executable)
index 0000000..9f73f02
--- /dev/null
@@ -0,0 +1,296 @@
+#!/usr/bin/env python
+# Copyright (C) 2012-2013, The CyanogenMod Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+
+import base64
+import json
+import netrc
+import os
+import re
+import sys
+try:
+  # For python3
+  import urllib.error
+  import urllib.parse
+  import urllib.request
+except ImportError:
+  # For python2
+  import imp
+  import urllib2
+  import urlparse
+  urllib = imp.new_module('urllib')
+  urllib.error = urllib2
+  urllib.parse = urlparse
+  urllib.request = urllib2
+
+from xml.etree import ElementTree
+
+product = sys.argv[1];
+
+if len(sys.argv) > 2:
+    depsonly = sys.argv[2]
+else:
+    depsonly = None
+
+try:
+    device = product[product.index("_") + 1:]
+except:
+    device = product
+
+if not depsonly:
+    print("Device %s not found. Attempting to retrieve device repository from CyanogenMod Github (http://github.com/CyanogenMod)." % device)
+
+repositories = []
+
+try:
+    authtuple = netrc.netrc().authenticators("api.github.com")
+
+    if authtuple:
+        githubauth = base64.encodestring('%s:%s' % (authtuple[0], authtuple[2])).replace('\n', '')
+    else:
+        githubauth = None
+except:
+    githubauth = None
+
+def add_auth(githubreq):
+    if githubauth:
+        githubreq.add_header("Authorization","Basic %s" % githubauth)
+
+if not depsonly:
+    githubreq = urllib.request.Request("https://api.github.com/search/repositories?q=%s+user:CyanogenMod+in:name+fork:true" % device)
+    add_auth(githubreq)
+    try:
+        result = json.loads(urllib.request.urlopen(githubreq).read().decode())
+    except urllib.error.URLError:
+        print("Failed to search GitHub")
+        sys.exit()
+    except ValueError:
+        print("Failed to parse return data from GitHub")
+        sys.exit()
+    for res in result.get('items', []):
+        repositories.append(res)
+
+local_manifests = r'.repo/local_manifests'
+if not os.path.exists(local_manifests): os.makedirs(local_manifests)
+
+def exists_in_tree(lm, path):
+    for child in lm.getchildren():
+        if child.attrib['path'] == path:
+            return True
+    return False
+
+# in-place prettyprint formatter
+def indent(elem, level=0):
+    i = "\n" + level*"  "
+    if len(elem):
+        if not elem.text or not elem.text.strip():
+            elem.text = i + "  "
+        if not elem.tail or not elem.tail.strip():
+            elem.tail = i
+        for elem in elem:
+            indent(elem, level+1)
+        if not elem.tail or not elem.tail.strip():
+            elem.tail = i
+    else:
+        if level and (not elem.tail or not elem.tail.strip()):
+            elem.tail = i
+
+def get_default_revision():
+    m = ElementTree.parse(".repo/manifest.xml")
+    d = m.findall('default')[0]
+    r = d.get('revision')
+    return r.replace('refs/heads/', '').replace('refs/tags/', '')
+
+def get_from_manifest(devicename):
+    try:
+        lm = ElementTree.parse(".repo/local_manifests/roomservice.xml")
+        lm = lm.getroot()
+    except:
+        lm = ElementTree.Element("manifest")
+
+    for localpath in lm.findall("project"):
+        if re.search("android_device_.*_%s$" % device, localpath.get("name")):
+            return localpath.get("path")
+
+    # Devices originally from AOSP are in the main manifest...
+    try:
+        mm = ElementTree.parse(".repo/manifest.xml")
+        mm = mm.getroot()
+    except:
+        mm = ElementTree.Element("manifest")
+
+    for localpath in mm.findall("project"):
+        if re.search("android_device_.*_%s$" % device, localpath.get("name")):
+            return localpath.get("path")
+
+    return None
+
+def is_in_manifest(projectpath):
+    try:
+        lm = ElementTree.parse(".repo/local_manifests/roomservice.xml")
+        lm = lm.getroot()
+    except:
+        lm = ElementTree.Element("manifest")
+
+    for localpath in lm.findall("project"):
+        if localpath.get("path") == projectpath:
+            return True
+
+    ## Search in main manifest, too
+    try:
+        lm = ElementTree.parse(".repo/manifest.xml")
+        lm = lm.getroot()
+    except:
+        lm = ElementTree.Element("manifest")
+
+    for localpath in lm.findall("project"):
+        if localpath.get("path") == projectpath:
+            return True
+
+    return False
+
+def add_to_manifest(repositories, fallback_branch = None):
+    try:
+        lm = ElementTree.parse(".repo/local_manifests/roomservice.xml")
+        lm = lm.getroot()
+    except:
+        lm = ElementTree.Element("manifest")
+
+    for repository in repositories:
+        repo_name = repository['repository']
+        repo_target = repository['target_path']
+        print('Checking if %s is fetched from %s' % (repo_target, repo_name))
+        if is_in_manifest(repo_target):
+            print('CyanogenMod/%s already fetched to %s' % (repo_name, repo_target))
+            continue
+
+        print('Adding dependency: CyanogenMod/%s -> %s' % (repo_name, repo_target))
+        project = ElementTree.Element("project", attrib = { "path": repo_target,
+            "remote": "github", "name": "CyanogenMod/%s" % repo_name })
+
+        if 'branch' in repository:
+            project.set('revision',repository['branch'])
+        elif fallback_branch:
+            print("Using fallback branch %s for %s" % (fallback_branch, repo_name))
+            project.set('revision', fallback_branch)
+        else:
+            print("Using default branch for %s" % repo_name)
+
+        lm.append(project)
+
+    indent(lm, 0)
+    raw_xml = ElementTree.tostring(lm).decode()
+    raw_xml = '<?xml version="1.0" encoding="UTF-8"?>\n' + raw_xml
+
+    f = open('.repo/local_manifests/roomservice.xml', 'w')
+    f.write(raw_xml)
+    f.close()
+
+def fetch_dependencies(repo_path, fallback_branch = None):
+    print('Looking for dependencies')
+    dependencies_path = repo_path + '/cm.dependencies'
+    syncable_repos = []
+
+    if os.path.exists(dependencies_path):
+        dependencies_file = open(dependencies_path, 'r')
+        dependencies = json.loads(dependencies_file.read())
+        fetch_list = []
+
+        for dependency in dependencies:
+            if not is_in_manifest(dependency['target_path']):
+                fetch_list.append(dependency)
+                syncable_repos.append(dependency['target_path'])
+
+        dependencies_file.close()
+
+        if len(fetch_list) > 0:
+            print('Adding dependencies to manifest')
+            add_to_manifest(fetch_list, fallback_branch)
+    else:
+        print('Dependencies file not found, bailing out.')
+
+    if len(syncable_repos) > 0:
+        print('Syncing dependencies')
+        os.system('repo sync --force-sync %s' % ' '.join(syncable_repos))
+
+    for deprepo in syncable_repos:
+        fetch_dependencies(deprepo)
+
+def has_branch(branches, revision):
+    return revision in [branch['name'] for branch in branches]
+
+if depsonly:
+    repo_path = get_from_manifest(device)
+    if repo_path:
+        fetch_dependencies(repo_path)
+    else:
+        print("Trying dependencies-only mode on a non-existing device tree?")
+
+    sys.exit()
+
+else:
+    for repository in repositories:
+        repo_name = repository['name']
+        if repo_name.startswith("android_device_") and repo_name.endswith("_" + device):
+            print("Found repository: %s" % repository['name'])
+            
+            manufacturer = repo_name.replace("android_device_", "").replace("_" + device, "")
+            
+            default_revision = get_default_revision()
+            print("Default revision: %s" % default_revision)
+            print("Checking branch info")
+            githubreq = urllib.request.Request(repository['branches_url'].replace('{/branch}', ''))
+            add_auth(githubreq)
+            result = json.loads(urllib.request.urlopen(githubreq).read().decode())
+
+            ## Try tags, too, since that's what releases use
+            if not has_branch(result, default_revision):
+                githubreq = urllib.request.Request(repository['tags_url'].replace('{/tag}', ''))
+                add_auth(githubreq)
+                result.extend (json.loads(urllib.request.urlopen(githubreq).read().decode()))
+            
+            repo_path = "device/%s/%s" % (manufacturer, device)
+            adding = {'repository':repo_name,'target_path':repo_path}
+            
+            fallback_branch = None
+            if not has_branch(result, default_revision):
+                if os.getenv('ROOMSERVICE_BRANCHES'):
+                    fallbacks = list(filter(bool, os.getenv('ROOMSERVICE_BRANCHES').split(' ')))
+                    for fallback in fallbacks:
+                        if has_branch(result, fallback):
+                            print("Using fallback branch: %s" % fallback)
+                            fallback_branch = fallback
+                            break
+
+                if not fallback_branch:
+                    print("Default revision %s not found in %s. Bailing." % (default_revision, repo_name))
+                    print("Branches found:")
+                    for branch in [branch['name'] for branch in result]:
+                        print(branch)
+                    print("Use the ROOMSERVICE_BRANCHES environment variable to specify a list of fallback branches.")
+                    sys.exit()
+
+            add_to_manifest([adding], fallback_branch)
+
+            print("Syncing repository to retrieve project.")
+            os.system('repo sync --force-sync %s' % repo_path)
+            print("Repository synced!")
+
+            fetch_dependencies(repo_path, fallback_branch)
+            print("Done")
+            sys.exit()
+
+print("Repository for %s not found in the CyanogenMod Github repository list. If this is in error, you may need to manually add it to your local_manifests/roomservice.xml." % device)