Run android integration tests in emulator by rPraml · Pull Request #2036 · mozilla/rhino

rPraml · 2025-08-23T20:04:28Z

The trigger was PR #2018 and #1152, which prompted me to revisit running Android tests in an emulator.
It’s still not fully finished, but I wanted to share the concept anyway.

In short: What does this PR

install-android-sdk.sh
downloads the SDK in RHINO_ROOT/android-sdk - if you do not already have one installed

it-android.sh
starts an android emulator, the emulator is accessible over http://localhost:6080

it-android
this project is a minimal android project, with embedded rhino (based on https://github.com/czak/minimal-android-project)

it executes all scripts in tests and outputs the result in a very minimalistic UI:

Open TODOs

make a testrunner similar to test262, so that it might run in CI (DONE!)
exclude the it-android, if no SDK is installed (DONE!)

@anivar I adapted your performance tests and I get ~297ms for 10000 iterations, with or without your fix. So there is no difference in the emulator. Do we perform the correct test? Or do I still not understand, what the problem is
(What I've understood: On native android (Dalvik/ART), the performance is worse, on a linux JVM, the performance is OK)

anivar · 2025-08-24T03:04:17Z

@rPraml I see you tested with and without the fix and got the same performance. This actually makes sense - the JIT compilation failure is environment-specific and typically occurs on:

Real Android devices (not emulators)
Older Android versions with stricter method size limits
ARM architectures vs x86/x64 emulators

The issue in #1365 was reported on real devices where the interpretLoop method exceeds the JIT threshold. Your emulator environment might have different or relaxed limits, which is why you're
seeing consistent good performance.

This reinforces why the fix is needed - it prevents the issue on affected devices while having no impact on environments that don't hit the threshold (like your emulator).

rPraml · 2025-08-24T10:33:42Z

Yes, until now I could not reproduce the issue.
Neither in the emulator nor on my real android device.

the JIT compilation failure is environment-specific

It seems that this bug is very tricky. Maybe I can retry on an older emulator version (although this would be x86)

This reinforces why the fix is needed

And it would be good, to have a test for this to avoid a regression in future releases, as the interpreter loop is code, that will be modified by a lot of people.

it prevents the issue on affected devices while having no impact on environments that don't hit the threshold (like your emulator).

@anivar I know, you are busy, but maybe you can provide details on which android device (android + SDK version) you were able to reproduce the issue. Maybe this is reproduceable in the emulator with same sdk (even if it is x86 and not arm)

rPraml

This is an initial (and working) solution for how Android tests could be executed in the github-CI pipeline.

The goal is to prevent regressions, which have repeatedly occurred in the past.

Tests can be easily added in the style of the ‘MozillaTestSuite’ as a JavaScript file in src/main/assets/tests, and an app (APK file) can also be built, which can be copied onto a phone to run the tests.

The CI-pipeline uses https://github.com/ReactiveCircus/android-emulator-runner

Additionally, with the budtmo/docker-android:emulator we have an easy to use solution for local debugging

I'd really appreciate community feedback on how we should proceed from here.

.github/workflows/gradle.yml

.gitignore

buildSrc/src/main/groovy/rhino.java-conventions.gradle

rPraml · 2025-08-24T16:30:25Z

it-android.sh

+#docker exec -it $CONTAINER_NAME adb install -r -t $APP && \
+#docker exec -it $CONTAINER_NAME adb shell am start -n com.example.rhino/com.example.rhino.MainActivity
+./gradlew it-android:connectedAndroidTest
+./gradlew it-android:installDebug


TODO: Would be nice, if someone else can confirm, these scripts will work

rPraml · 2025-08-24T16:31:02Z

it-android/build.gradle

+}
+
+if (!System.getenv("ANDROID_HOME") && !localProperties.get("sdk.dir")) {
+    System.out.println("No Android SDK found. Skipping build")


Skip android build for users, that do not have a SDK installed

it-android/build.gradle

rPraml · 2025-08-24T16:33:58Z

it-android/src/main/assets/assert.js

@@ -0,0 +1,417 @@
+// Copyright 2008 the V8 project authors. All rights reserved.


This is a 1:1 copy of assert.js from MozillaTestSuite

it-android/src/androidTest/java/org/mozilla/javascript/android/test/RhinoTest.java

rPraml · 2025-08-24T16:35:06Z

it-android/src/main/java/org/mozilla/javascript/android/TestCase.java

+            new ContextFactory() {
+                @Override
+                protected boolean hasFeature(Context cx, int featureIndex) {
+                    if (featureIndex == Context.FEATURE_ENABLE_XML_SECURE_PARSING) return false;


Secure-parsing cannot be changed on android-devices

anivar · 2025-08-24T17:21:39Z

@rPraml Thanks for this Android testing infrastructure!

The JIT issue from #2018 occurs when template literals generate methods exceeding ~7392KB, causing ART compiler failures on certain Android devices.

Key factors:

Android 8-10 versions most affected
Device-specific memory/CPU constraints
Hard to reproduce in emulators

Suggestions for your framework:

Test multiple API levels (26, 28, 30)
Add bytecode size monitoring
Include ARM architecture in CI matrix

The fix prevents the issue, but automated tests will help catch regressions.

anivar · 2025-08-24T17:28:01Z

@rPraml Great work on the implementation details! Here's feedback on your specific questions:

KVM rules (gradle.yml): Yes, needed for hardware acceleration in GitHub Actions runners. Without it, emulators run extremely slow.

Build integration: Run as a separate matrix job, not part of Java 21 build. This allows:

Parallel execution
Clear failure isolation
Easier to skip for non-Android changes

SDK installation script: Keep it. Not everyone has Android SDK locally, and it ensures consistent SDK versions across contributors.

API Level 33: While Play Store requires API 33, for testing we should include older versions too:

minSdk = 26  // Android 8.0 where JIT issues start
targetSdk = 33

Architecture suggestion: Add ARM to your emulator matrix since x86_64 won't catch ARM-specific JIT issues:

strategy:
  matrix:
    api-level: [26, 28, 30, 33]
    arch: [x86_64, arm64-v8a]

The local Docker approach is excellent for debugging. Overall, this is a solid foundation!

gbrail · 2025-08-24T21:30:36Z

This is great progress -- I know it's a pain to get this in.

I think that we should create a separate GitHub Action for Android, rather than putting it in the main tests. That might make it easier to develop, and also it can run in parallel. We still won't merge anything until all the checks have passed.

Also, I personally don't use Docker, but Podman, and I wonder if others do too? If so I may try to adapt the script (should just be replacing "docker" with "podman").

821938089 · 2025-08-25T04:33:24Z

minsdk should be 21, 26 is too high.

821938089 · 2025-08-25T08:18:21Z

CI failure under minSdk 21 is a regression introduced by PR #1785.
For details, see #1785 (comment).

Switching the minSDK back to 26 is not the correct solution, as this will prevent Android projects with a minSDK of 21 from compiling, resulting in the same failure as the CI.

rPraml · 2025-08-26T07:29:34Z

This task gave me a much better understanding of how Android works under the hood.

(For example, I wasn’t aware that the entire rhino.jar is converted by dx into a completely different Dalvik bytecode format and isn’t bundled into the .apk as a regular .jar file.)

Summary

The goal of this pull request is to provide the infrastructure for Android tests (we are not aiming to fix any bugs at this stage.)

✅ Tests run successfully for API levels 26, 28, 30, and 33 on x86_64
❌ Tests for arm64-v8a have not yet been made to run as a GitHub Action
❌ Tests for API level 21 are currently not possiblel, since PR #1785 introduced some incompatibilities
❌ I haven’t been able to reproduce bug #2018 so far

Next Steps

Over the next few days, I’ll improve the documentation and related materials to establish a fully functional Android testing infrastructure as quickly as possible.
Hopefully someone will be able to reproduce bug Fix Android JIT compilation failure with template literals #2018 (possibly by running with less RAM or other system limitations, though @anivar mentioned this is likely very hard to reproduce in an emulator).
We need to decide on the minimum supported API level (minSdkVersion).
Thanks to @821938089 for pointing to the right direction
@aardvark179, @rbri, @gbrail:
I reviewed the changes introduced in Thread safe slot maps without containers #1785. It really only affects the ThreadedAccess class.
Although @SuppressWarnings("AndroidJdkLibsChecker") is used, accessing VarHandle.compareAndExchange causes a compilation error with dx, meaning Rhino can no longer be used on older Android versions.
I see the following options here:
- We are defining API level 26 as the minSdkVersion
- We find some workaround for the ThreadedAccess issue (e.g. filter this class in dexing process - I already tried a proguard-rule, but I wasn’t able to make it work) - with desugaring, we could at least achive compatibility with API 21 (See [Experimental] Testing older android APIs #2042)
- Remove all non-android-libs from the core rhino module and provide them with a service. So we will get a rhino-jvm and and maybe a rhino-dalvik module, that provides different impls for ThreadedAccess
And last but not least, some good news: Due my reasearch, I found an interesting project: https://github.com/linkedin/dexmaker It should be possible to generate dalvik bytecode.

anivar · 2025-08-26T12:51:26Z

@rPraml Thank you for this comprehensive Android testing infrastructure! Your deep dive into the dx/Dalvik bytecode conversion and the testing setup is excellent work.

I've been thinking about the DexMaker suggestion and wondering if there might be alternative approaches that better align with Rhino's strengths and modern Android development:

On DexMaker:
While DexMaker (Apache 2.0) is license-compatible with Rhino (MPL 2.0), it primarily addresses the challenge of runtime bytecode generation on Android. Note that only dexmaker-mockito-inline requires API 28+ - the core library supports older versions. Still, I wonder if the added complexity and APK size increase are justified given that most Rhino Android use cases might not need dynamic compilation?

Alternative Approaches to Consider:

Kotlin Multiplatform - Could allow sharing code between JVM and Android while keeping Rhino's pure-Java nature
GraalVM Native Image - Experimental Android support could eliminate interpreter overhead entirely
Focus on Rhino's unique strengths - Rather than competing on performance, optimize for use cases where Rhino excels:
- Testing frameworks that need deep Java mocking
- Build tools and automation scripts
- Enterprise apps with existing Rhino dependencies
- Educational environments where Java integration matters

On the Module Split:

Given that Dalvik was replaced by ART in Android 5.0 (2014), perhaps a simpler approach would be a single module with runtime feature detection rather than rhino-jvm/rhino-dalvik?

On Minimum API Level:
With Google Play requiring API 35 for new apps (as of August 31, 2025), setting minSdkVersion to 24 (Android 7.0) might be pragmatic - this would resolve the VarHandle issues while covering the vast majority of devices.

Your testing infrastructure gives us exactly what we need to make these decisions based on real data. The ability to run tests across multiple API levels is invaluable.

What do you think about focusing the Android strategy on Rhino's unique advantages (pure Java, no JNI, excellent Java interop) rather than trying to match the performance of V8 or QuickJS?

gbrail

Thanks! I get where this is going and it is going to help a lot.

So I can understand -- if I run the "install-android-sdk" script, and then run the tests, will it use the SDK to run the tests on Android, or do I need to do the whole docker / podman setup for that?

I'm asking because installing the SDK works great, and the tests work fine in either way, but getting that docker / podman script to work portably across OSse and other things is going to take some work -- I had to muck around to make it work with Windows and MSYS but there are probably five other combinations that people use. Do we get some value out of this if we don't need people to run the container, but if the CI job does?

gbrail · 2025-08-29T22:17:33Z

install-android-sdk.sh

@@ -0,0 +1,15 @@
+#/usr/bin/env bash


Can you please move this file to the "it-android" directory so that the root directory doesn't get too full?

Could you also make it executable (might need to mess around with Git -- usually if you "chmod a+x" the file Git will figure it out when you first add it).

gbrail · 2025-08-29T22:18:01Z

run-android-tests-locally.sh

@@ -0,0 +1,40 @@
+#/usr/bin/env bash


Could you also move this file to it-android and make it executable?

> Could not resolve all files for configuration ':it-android:debugRuntimeClasspath'. > Failed to transform rhino-1.8.1-SNAPSHOT.jar (project :rhino) to match attributes {artifactType=android-dex, dexing-component-attributes=ComponentSpecificParameters(minSdkVersion=21, debuggable=true, enableCoreLibraryDesugaring=false, enableGlobalSynthetics=false, enableApiModeling=false, dependenciesClassesAreInstrumented=false, asmTransformComponent=null, useJacocoTransformInstrumentation=false, enableDesugaring=true, needsClasspath=true, useFullClasspath=false, componentIfUsingFullClasspath=null), org.gradle.category=library, org.gradle.dependency.bundling=external, org.gradle.jvm.version=11, org.gradle.libraryelements=jar, org.gradle.usage=java-runtime}. > Execution failed for DexingWithClasspathTransform: /home/runner/work/rhino/rhino/rhino/build/libs/rhino-1.8.1-SNAPSHOT.jar. > Error while dexing. Increase the minSdkVersion to 26 or above.

rPraml · 2025-10-07T07:28:16Z

tests/test262

This has to be reverted!

gbrail · 2025-10-29T02:30:20Z

I'm going to close this PR as I've included all the commits in #2146 and I would like to merge that soon. Thanks!

anivar mentioned this pull request Aug 24, 2025

Fix Android JIT compilation failure with template literals #2018

Closed

rPraml force-pushed the android-it-tests branch 2 times, most recently from 9c60cf6 to 1b586ee Compare August 24, 2025 14:36

rPraml mentioned this pull request Aug 24, 2025

Create an Android regression test #1152

Open

rPraml commented Aug 24, 2025

View reviewed changes

rPraml marked this pull request as ready for review August 24, 2025 16:50

rPraml mentioned this pull request Aug 24, 2025

Made spotless as separate convention #2039

Merged

rPraml force-pushed the android-it-tests branch 3 times, most recently from 1a8e208 to a799cb8 Compare August 24, 2025 20:01

gbrail reviewed Aug 29, 2025

View reviewed changes

rPraml mentioned this pull request Sep 25, 2025

The performance of version 1.7.14 has gone backwards #1365

Open

rPraml force-pushed the android-it-tests branch from 7ae58e3 to c858a5d Compare September 30, 2025 06:34

rPraml added 8 commits October 6, 2025 20:09

Add SDK download script

73bf06c

Test in emulator works

f1e081a

Made SDK optional

312a2da

Spotless

bf28504

Try older SDK and try to understand ART

2aa838e

Add GH workflow

1b6b8c8

Add real testcase

8466f39

Spotless

b1c90a7

rPraml and others added 11 commits October 6, 2025 20:09

fixed sealed shared context issue

3bedbbe

Linter complains about API level

2df3f24

Test matrix build

d087994

Try to add emulator.log

3e84630

Try to add arm platform

96e0f89

Disabled ARM and change minSdk to 21

7aa1111

Let's focus on working stuff

90f1ad5

Provide documentation

8886857

Add compatibility for older rhino versions

c96785a

Add test for mozilla#1365

385674e

rPraml force-pushed the android-it-tests branch from 41b31a2 to 385674e Compare October 6, 2025 18:09

rPraml added 2 commits October 6, 2025 21:23

Performancetests

da13905

Spotless

f5aa0bd

rPraml commented Oct 7, 2025

View reviewed changes

tests/test262

Copy link
Copy Markdown

Contributor Author

rPraml Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has to be reverted!

gbrail closed this Oct 29, 2025

gbrail mentioned this pull request Oct 29, 2025

Re-create test framework for android #2146

Merged

		@@ -0,0 +1,417 @@
		// Copyright 2008 the V8 project authors. All rights reserved.

Conversation

rPraml commented Aug 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anivar commented Aug 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rPraml commented Aug 24, 2025

Uh oh!

rPraml left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rPraml Aug 24, 2025

Choose a reason for hiding this comment

Uh oh!

rPraml Aug 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rPraml Aug 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rPraml Aug 24, 2025

Choose a reason for hiding this comment

Uh oh!

anivar commented Aug 24, 2025

Uh oh!

anivar commented Aug 24, 2025

Uh oh!

gbrail commented Aug 24, 2025

Uh oh!

821938089 commented Aug 25, 2025

Uh oh!

821938089 commented Aug 25, 2025

Uh oh!

rPraml commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anivar commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gbrail left a comment

Choose a reason for hiding this comment

Uh oh!

gbrail Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

gbrail Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

rPraml Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

gbrail commented Oct 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rPraml commented Aug 23, 2025 •

edited

Loading

anivar commented Aug 24, 2025 •

edited

Loading

rPraml commented Aug 26, 2025 •

edited

Loading

anivar commented Aug 26, 2025 •

edited

Loading