Firebase Device Farm / Firebase Test Labs
Basically, Firebase device farm is nothing but it allows us to test our app on a variety of devices.
In case of limitation of the device, it’s not possible to test with each device and for that Firebase device farm comes into the picture.
Firebase Device Farm or Firebase test labs provide us with devices which are hosted on Google Cloud.
You can choose a device according to your requirement if that device is not available you can request a device with details of the required configuration.
Testing quotas / Pricing
Firebase Test Lab provides a Cloud API quota and a testing quota, which is included in the standard Spark and Blaze pricing plans.
- Spark plan (no-cost): The resource limits are listed for up to 15 test runs per day in total:
- 10 test runs per day on virtual devices (Mostly it is for Android)
- 5 test runs per day on physical devices
- Blaze plan: Not all projects in the Blaze plan have the same quotas. If your usage of Google Cloud increases over time, your quotas might increase correspondingly.
- If you expect a sizable upcoming increase in usage, you can proactively request for adjusting quota by submitting an increasing request or contacting support.
Billing is calculated by the minutes spent running tests. The Blaze plan begins with a no-cost time limit that’s similar to the resource limit offered by the Spark plan: - 30 minutes of test time per day on physical devices
- 60 minutes of test time per day on virtual devices
- Any usage above these limits is charged according to the following hourly rates:
- $5 per hour for each physical device.
- $1 per hour for each virtual device.
Charges are calculated on a per-minute basis, rounded up to the nearest minute. For example, a 22-second test is billed for one minute, while a 75-second test is billed for two minutes.
You are charged only for the time spent running tests (the time it takes to install your app and collect test results will not be charged).
Test Lab quotas are applied project-level, not site-level. These limits are shared across all test types (instrumentation, Robo, and Game Loop) and test matrices. When you run a test, you can check its run time (i.e., the time it takes the test to run) under Test execution and test matrix results in the Firebase console. Run times are displayed next to each device. If you’re in the Blaze plan, the test run times are used for billing.
Tools
We can choose one tool out of the following.
- Test with the Firebase console — The Firebase console lets you upload an app and initiate testing from anywhere
- Test with the gcloud CLI — Enables you to run tests from the command line interactively, and is also well suited for scripting as part of your automated build and testing process.
Note that all test types can run up to a maximum of 45 minutes on physical devices, and any uncaught exception will cause a test failure
Type of Tests For iOS
- Robo Test (Beta Version is available for iOS) — The Robo test automatically explores your app on a wide array of devices to find defects and report any crashes that occur. Robo test doesn’t require you to write app tests
- Game Loop Test — A Game Loop runs your app by launching scenarios through a custom URL scheme
- XCTest — Create and run unit tests, performance tests, and UI tests for your Xcode project. (Written by team)
Types of Tests for Android
- Robo Test -
- Game Loop Test -
- Instrumentation — Written by you or by the team using Espresso or UI Automator Android Test framework
Implementation
Steps
- Create a Project on the Firebase console
- Configure / Customize derived data location.
- Build test for generic device Prodcut->Build for->Testing
- Go to the configured derived data location select both the Debug-iphoneos and .xctesrun folder and compress it.
- Open the Project you created on the Firebase console and add the zip file under Test Lab section.
- Select your Xcode version from drop down.
- Click continue and select device dimensions(Which nothing but the available devices on gcloud center)
- After some time it returns test results or whatever error occurred during test case.
Questions
Is there any difference in pricing for Android and iOS?
No, There is no difference between Android and iOS pricing.
What is Virtual Device and Physical Device?
A virtual device is a Simulator/Emulator and the Physical device is a real device hosted on the Google Cloud center.
How the time is calculated?
It only considers the time which is taken to complete the test. The time taken by the app to install and the time taken to collect test results is ignored.
What is the maximum free test run time?
30 minutes of test time per day on physical devices.
60 minutes of test time per day on virtual devices.
Usage above this is chargeable.
For how long tests can be run?
Note that all test types collectively can run up to a maximum of 45 minutes on physical devices, and any uncaught exception will cause a test failure.
Can I request a device with a particular configuration?
We have a list of available devices and in case your required device is not available in devices then you can request a device with all the details asked.
Is requesting a particular device chargeable?
No.
