Surface API 33-37 and 16 KB-page system images

README.md

@@ -146,38 +146,32 @@ of:
 - Available and currently Docker-compatible system images
 - Currently published and advertised emulator binaries
 
-For each system image, the API level, variant, ABI, and URL are displayed. For
-each emulator, the update channel (stable vs canary), version, host os, and URL
+For each system image, the codename letter, sort tag, ABI, API level (with a
+trailing `ps16k` for 16 KB-page variants), and URL are displayed. For each
+emulator, the update channel (stable, dev, ...), version, host os, and URL
 are displayed.
 
 Example output:
 
-    SYSIMG android 21 L x86_64 https://dl.google.com/android/repository/sys-img/android/x86_64-21_r05.zip
-    SYSIMG android 22 L x86_64 https://dl.google.com/android/repository/sys-img/android/x86_64-22_r06.zip
-    SYSIMG android 23 M x86_64 https://dl.google.com/android/repository/sys-img/android/x86_64-23_r10.zip
-    SYSIMG android 24 N x86_64 https://dl.google.com/android/repository/sys-img/android/x86_64-24_r08.zip
-    SYSIMG android 25 N x86_64 https://dl.google.com/android/repository/sys-img/android/x86_64-25_r01.zip
-    SYSIMG android 26 O x86_64 https://dl.google.com/android/repository/sys-img/android/x86_64-26_r01.zip
-    SYSIMG android 27 O x86_64 https://dl.google.com/android/repository/sys-img/android/x86_64-27_r01.zip
-    SYSIMG android 28 P x86_64 https://dl.google.com/android/repository/sys-img/android/x86_64-28_r04.zip
-    SYSIMG android 28 Q x86_64 https://dl.google.com/android/repository/sys-img/android/x86_64-Q_r04.zip
-    SYSIMG google_apis 21 L x86_64 https://dl.google.com/android/repository/sys-img/google_apis/x86_64-21_r30.zip
-    SYSIMG google_apis 22 L x86_64 https://dl.google.com/android/repository/sys-img/google_apis/x86_64-22_r24.zip
-    SYSIMG google_apis 23 M x86_64 https://dl.google.com/android/repository/sys-img/google_apis/x86_64-23_r31.zip
-    SYSIMG google_apis 24 N x86_64 https://dl.google.com/android/repository/sys-img/google_apis/x86_64-24_r25.zip
-    SYSIMG google_apis 25 N x86_64 https://dl.google.com/android/repository/sys-img/google_apis/x86_64-25_r16.zip
-    SYSIMG google_apis 26 O x86_64 https://dl.google.com/android/repository/sys-img/google_apis/x86_64-26_r13.zip
-    SYSIMG google_apis 28 P x86_64 https://dl.google.com/android/repository/sys-img/google_apis/x86_64-28_r09.zip
-    SYSIMG google_apis 28 Q x86_64 https://dl.google.com/android/repository/sys-img/google_apis/x86_64-Q_r04.zip
-    SYSIMG google_apis_playstore 28 P x86_64 https://dl.google.com/android/repository/sys-img/google_apis_playstore/x86_64-28_r08.p
-    SYSIMG google_apis_playstore 28 Q x86_64 https://dl.google.com/android/repository/sys-img/google_apis_playstore/x86_64-Q_r04.zp
-    EMU stable 29.0.11 windows https://dl.google.com/android/repository/emulator-windows-5598178.zip
-    EMU stable 29.0.11 macosx https://dl.google.com/android/repository/emulator-darwin-5598178.zip
-    EMU stable 29.0.11 linux https://dl.google.com/android/repository/emulator-linux-5598178.zip
-    EMU stable 28.0.25 windows https://dl.google.com/android/repository/emulator-windows-5395263.zip
-    EMU canary 29.0.12 windows https://dl.google.com/android/repository/emulator-windows-5613046.zip
-    EMU canary 29.0.12 macosx https://dl.google.com/android/repository/emulator-darwin-5613046.zip
-    EMU canary 29.0.12 linux https://dl.google.com/android/repository/emulator-linux-5613046.zip
+    SYSIMG R google_apis x86_64 30 https://dl.google.com/android/repository/sys-img/google_apis/x86_64-30_r16.zip
+    SYSIMG S google_apis x86_64 31 https://dl.google.com/android/repository/sys-img/google_apis/x86_64-31_r14.zip
+    SYSIMG T google_apis_playstore x86_64 33 https://dl.google.com/android/repository/sys-img/google_apis_playstore/x86_64-33_r09.zip
+    SYSIMG U google_apis x86_64 34 https://dl.google.com/android/repository/sys-img/google_apis/x86_64-34_r14.zip
+    SYSIMG V google_apis x86_64 35 https://dl.google.com/android/repository/sys-img/google_apis/x86_64-35_r09.zip
+    SYSIMG V google_apis x86_64 35 ps16k https://dl.google.com/android/repository/sys-img/google_apis/x86_64-ps16k-35_r05.zip
+    SYSIMG B google_apis x86_64 36 https://dl.google.com/android/repository/sys-img/google_apis/x86_64-36_r07.zip
+    SYSIMG B google_apis x86_64 36 ps16k https://dl.google.com/android/repository/sys-img/google_apis/x86_64-ps16k-36_r07.zip
+    SYSIMG B google_apis_playstore x86_64 36 https://dl.google.com/android/repository/sys-img/google_apis_playstore/x86_64-36_r07.zip
+    SYSIMG B google_apis_playstore x86_64 36 ps16k https://dl.google.com/android/repository/sys-img/google_apis_playstore/x86_64-playstore-ps16k-36_r07.zip
+    SYSIMG C google_apis x86_64 37.0 ps16k https://dl.google.com/android/repository/sys-img/google_apis/x86_64-ps16k-37.0_r04.zip
+    EMU stable 36.5.11 linux https://dl.google.com/android/repository/emulator-linux_x64-15261927.zip
+    EMU dev 36.6.8 linux https://dl.google.com/android/repository/emulator-linux_x64-15368433.zip
+
+A `ps16k` suffix marks a 16 KB page-size variant of the same `(api, sort, abi)`;
+the regular 4 KB variant continues to appear without the suffix. Pre-release
+codename builds (Baklava, CinnamonBun, ...) and ext-SDK images are filtered out
+to keep the list uncluttered; pass them as a direct zip path to
+`emu-docker create` if you need to build a container from one.
 
 One can then use tools like `wget` or a browser to download a desired emulator
 and system image. After the two are obtained, we can build a Docker image.

emu/android_release_zip.py

@@ -181,6 +181,7 @@ def __init__(self, file_name: str):
         self.props["qemu.tag"] = self.tag()
         self.props["qemu.short_tag"] = self.short_tag()
         self.props["qemu.short_abi"] = self.short_abi()
+        self.props["qemu.is_16k"] = "true" if self.is_16k() else "false"
 
     def api(self) -> str:
         """The api level, if any."""
@@ -211,12 +212,24 @@ def gpu(self) -> str:
         return self.props.get("SystemImage.GpuSupport")
 
     def tag(self) -> str:
-        """The tag associated with this release."""
-        tag = self.props.get("SystemImage.TagId", "")
+        """The canonical sort tag associated with this release.
+
+        SystemImage.TagId can be multi-valued (comma-joined) for 16 KB-page
+        variants -- typically "google_apis,page_size_16kb". Reduce to the
+        non-page-size component so downstream lookups still work.
+        """
+        raw = self.props.get("SystemImage.TagId", "")
+        parts = [p for p in raw.split(",") if p and p != "page_size_16kb"]
+        tag = parts[0] if parts else ""
         if tag == "default" or tag.strip() == "":
             tag = "android"
         return tag
 
+    def is_16k(self) -> bool:
+        """True if this is a 16 KB page-size system image variant."""
+        raw = self.props.get("SystemImage.TagId", "")
+        return "page_size_16kb" in raw.split(",")
+
     def short_tag(self) -> str:
         """A shorthand tag."""
         return self.SHORT_TAG[self.tag()]

emu/containers/emulator_container.py

@@ -105,6 +105,8 @@ def image_name(self):
             self.props["qemu.short_tag"],
             self.props["qemu.short_abi"],
         )
+        if self.props.get("qemu.is_16k") == "true":
+            name = "{}-ps16k".format(name)
         if not self.metrics:
             return "{}-no-metrics".format(name)
         return name

emu/containers/system_image_container.py

@@ -64,7 +64,8 @@ def image_name(self):
         if self.system_image_info:
             return self.system_image_info.image_name()
         if self.system_image_zip:
-            return f"sys-{self.system_image_zip.api()}-{self.system_image_zip.short_tag()}-{self.system_image_zip.short_abi()}"
+            suffix = "-ps16k" if self.system_image_zip.is_16k() else ""
+            return f"sys-{self.system_image_zip.api()}-{self.system_image_zip.short_tag()}-{self.system_image_zip.short_abi()}{suffix}"
 
     def docker_tag(self):
         if self.system_image_zip:

emu/emu_downloads_menu.py

@@ -28,11 +28,11 @@
 from emu.docker_config import DockerConfig
 
 SYSIMG_REPOS = [
-    "https://dl.google.com/android/repository/sys-img/android/sys-img2-1.xml",
-    "https://dl.google.com/android/repository/sys-img/google_apis/sys-img2-1.xml",
-    "https://dl.google.com/android/repository/sys-img/google_apis_playstore/sys-img2-1.xml",
-    "https://dl.google.com/android/repository/sys-img/google_atd/sys-img2-1.xml",
-    "https://dl.google.com/android/repository/sys-img/android-tv/sys-img2-1.xml",
+    "https://dl.google.com/android/repository/sys-img/android/sys-img2-5.xml",
+    "https://dl.google.com/android/repository/sys-img/google_apis/sys-img2-5.xml",
+    "https://dl.google.com/android/repository/sys-img/google_apis_playstore/sys-img2-5.xml",
+    "https://dl.google.com/android/repository/sys-img/google_atd/sys-img2-5.xml",
+    "https://dl.google.com/android/repository/sys-img/android-tv/sys-img2-5.xml",
 ]
 
 EMU_REPOS = ["https://dl.google.com/android/repository/repository2-1.xml"]
@@ -63,11 +63,17 @@
     "30": "R",
     "31": "S",
     "32": "S",
+    "33": "T",
+    "34": "U",
+    "35": "V",
+    "36": "B",
+    "37": "C",
 }
 
-# Older versions might not work as expected.
-MIN_REL_I386 = "K"
-MIN_REL_X64 = "O"
+# Older versions might not work as expected. Keyed on the major API
+# integer so new API levels need no code change to be filtered correctly.
+MIN_API_I386 = 19  # K
+MIN_API_X64 = 26   # O
 
 
 class License(object):
@@ -137,21 +143,43 @@ class SysImgInfo(LicensedObject):
     def __init__(self, pkg, licenses):
         super(SysImgInfo, self).__init__(pkg, licenses)
         details = pkg.find("type-details")
+        # api-level is "36", "36.1", "37.0", or "36x" (ext-SDK). Keep the
+        # original string for display/uniqueness and derive a major int
+        # for filter thresholds.
         self.api = details.find("api-level").text
-
+        m = re.match(r"\d+", self.api)
+        self.api_major = int(m.group()) if m else 0
+
+        # Derive the (sort, is_16k) pair from the path's third segment
+        # (system-images;<api_or_codename>;<sort>;<abi>). Tag elements in
+        # the XML can be multi-valued and ordered arbitrarily, so the
+        # path is the only deterministic source.
+        path_parts = pkg.attrib.get("path", "").split(";")
+        sort = path_parts[2] if len(path_parts) > 2 else ""
+        self.is_16k = sort.endswith("_ps16k")
+        sort_base = sort[: -len("_ps16k")] if self.is_16k else sort
+
+        # Pre-release builds carry a codename (Baklava, CinnamonBun, CANARY,
+        # ...). They get filtered out at listing time so users only see
+        # stable/released images by default.
         codename = details.find("codename")
-        if codename is None:
-            if self.api in API_LETTER_MAPPING:
-                self.letter = API_LETTER_MAPPING[self.api]
-            else:
-                self.letter = "A"  # A indicates unknown code.
+        self.is_preview = codename is not None and bool(codename.text)
+        # ext-SDK system images (api-level "33x", "34x", "36x", ...) ship
+        # the same target SDK with additional extension APIs. They are
+        # filtered out at listing time to keep the list uncluttered.
+        self.is_ext_sdk = self.api.endswith("x")
+        if self.api in API_LETTER_MAPPING:
+            self.letter = API_LETTER_MAPPING[self.api]
+        elif str(self.api_major) in API_LETTER_MAPPING:
+            self.letter = API_LETTER_MAPPING[str(self.api_major)]
         else:
-            self.letter = codename.text
-
-        self.tag = details.find("tag").find("id").text
+            self.letter = self.api
 
-        if self.tag == "default":
-            self.tag = "android"
+        # The legacy sort segment "default" is served from the "android"
+        # subdirectory and surfaced to users as the "android" tag.
+        if sort_base == "default":
+            sort_base = "android"
+        self.tag = sort_base or details.find("tag").find("id").text
         self.abi = details.find("abi").text
 
         # prefer a url for a Linux host in case there are multiple
@@ -161,35 +189,45 @@ def __init__(self, pkg, licenses):
             url_element = pkg.find(".//url")
         self.zip = url_element.text
 
+        # The zip lives under sys-img/<sort_base>/, regardless of how
+        # the <tag> element is labelled — for 16KB variants the path
+        # sort is e.g. google_apis_ps16k, served from .../google_apis/.
+        url_dir = sort_base or self.tag
         self.url = "https://dl.google.com/android/repository/sys-img/%s/%s" % (
-            self.tag,
+            url_dir,
             self.zip,
         )
 
     def short_tag(self):
-        return self.SHORT_TAG[self.tag]
+        return self.SHORT_TAG.get(self.tag, self.tag)
 
     def short_abi(self):
-        return self.SHORT_MAP[self.abi]
+        return self.SHORT_MAP.get(self.abi, self.abi)
 
     def image_name(self):
-        return "sys-{}-{}-{}".format(self.api, self.short_tag(), self.short_abi())
+        suffix = "-ps16k" if self.is_16k else ""
+        return "sys-{}-{}-{}{}".format(
+            self.api, self.short_tag(), self.short_abi(), suffix
+        )
 
     def download_name(self):
-        return "sys-img-{}-{}-{}-{}.zip".format(
-            self.tag, self.api, self.letter, self.abi
+        suffix = "-ps16k" if self.is_16k else ""
+        return "sys-img-{}-{}-{}-{}{}.zip".format(
+            self.tag, self.api, self.letter, self.abi, suffix
         )
 
     def download(self, dest=Path.cwd()):
         dest = dest / self.download_name()
+        variant = " ps16k" if self.is_16k else ""
         print(
-            f"Downloading system image: {self.tag} {self.api} {self.letter} {self.abi} to {dest}"
+            f"Downloading system image: {self.tag} {self.api} {self.letter} {self.abi}{variant} to {dest}"
         )
 
         return super(SysImgInfo, self).download(self.url, dest)
 
     def __str__(self):
-        return "{} {} {}".format(self.letter, self.tag, self.abi)
+        suffix = " ps16k" if self.is_16k else ""
+        return "{} {} {}{}".format(self.letter, self.tag, self.abi, suffix)
 
 
 class EmuInfo(LicensedObject):
@@ -246,17 +284,24 @@ def get_images_info(arm=False):
     xml = [ET.fromstring(x).findall("remotePackage") for x in xml]
     # Flatten the list of lists into a system image objects.
     infos = [SysImgInfo(item, licenses) for sublist in xml for item in sublist]
-    # Filter only for intel images that we know that work
+    # Drop pre-release builds (Baklava / CinnamonBun / CANARY etc.) and
+    # ext-SDK images (api-level "36x" etc.) to keep the list uncluttered.
+    infos = [info for info in infos if not info.is_preview and not info.is_ext_sdk]
+    # Filter only for intel images that we know that work. Filtering on
+    # the integer api_major so new API levels don't need a code change.
     x86_64_imgs = [
-        info for info in infos if info.abi == "x86_64" and info.letter >= MIN_REL_X64
+        info for info in infos if info.abi == "x86_64" and info.api_major >= MIN_API_X64
     ]
     x86_imgs = [
-        info for info in infos if info.abi == "x86" and info.letter >= MIN_REL_I386
+        info for info in infos if info.abi == "x86" and info.api_major >= MIN_API_I386
     ]
     slow = []
     if arm:
         slow = [info for info in infos if info.abi.startswith("arm")]
-    all_imgs = sorted(x86_64_imgs + x86_imgs + slow, key=lambda x: x.api + x.tag)
+    all_imgs = sorted(
+        x86_64_imgs + x86_imgs + slow,
+        key=lambda x: (x.api_major, x.api, x.tag, x.abi, x.is_16k),
+    )
     # Filter out windows/darwin images.
     return [i for i in all_imgs if "windows" not in i.url and "darwin" not in i.url]
 
@@ -332,7 +377,7 @@ def select_image(arm):
     Returns a SysImgInfo object with the choice or None if the user aborts."""
     img_infos = get_images_info(arm)
     display = [
-        f"{img_info.api} {img_info.letter} {img_info.tag} ({img_info.abi})"
+        f"{img_info.api} {img_info.letter} {img_info.tag} ({img_info.abi}){' ps16k' if img_info.is_16k else ''}"
         for img_info in img_infos
     ]
 
@@ -360,9 +405,15 @@ def list_all_downloads(arm):
     emu_infos = get_emus_info()
 
     for img_info in img_infos:
+        variant = " ps16k" if img_info.is_16k else ""
         print(
-            "SYSIMG {} {} {} {} {}".format(
-                img_info.letter, img_info.tag, img_info.abi, img_info.api, img_info.url
+            "SYSIMG {} {} {} {}{} {}".format(
+                img_info.letter,
+                img_info.tag,
+                img_info.abi,
+                img_info.api,
+                variant,
+                img_info.url,
             )
         )
 

emu/templates/Dockerfile.system_image

@@ -48,7 +48,8 @@ LABEL maintainer="{{user}}" \
     qemu.tag="{{qemu_tag}}" \
     qemu.cpu="{{qemu_cpu}}" \
     qemu.short_tag="{{qemu_short_tag}}" \
-    qemu.short_abi="{{qemu_short_abi}}"
+    qemu.short_abi="{{qemu_short_abi}}" \
+    qemu.is_16k="{{qemu_is_16k}}"
 
 # We adopt the following naimg convention <ro.build.version.sdk>-<qem>-<ro.product.cpu.abi>
 # SystemImage.TagId in 'aosp', 'google', 'playstore'ß