Download to read offline
Uploaded byAnand Bagmar
Streamlining End-to-End Testing Automation
The document discusses the complexities and challenges of end-to-end test automation in distributed teams, particularly in hybrid working environments with varied system setups. It outlines solutions for consistently setting up test environments, managing dependencies, and executing tests across multiple platforms using the 'teswiz' framework, including automation for CI/CD pipelines. Additionally, it emphasizes the importance of establishing a 'hard gate' for tests to ensure product quality and integrates AI for enhanced validation and reporting.
More Related Content
Similar to Streamlining End-to-End Testing Automation
Streamlining End-to-End Testing Automation
- 1.
- 2.
- 3. Ground Reality Distributed teams Hybrid working Differentnetwork setup and speeds Many teams (>100) Mac, Windows and Linux laptops Different Software versions Certificates, Policies and multiple VPNs Inconsistent Developer & SDET experience Test execution environment setup is tedious System Tests (e2e & component UI): teswiz (Appium, Selenium, Applitools, JDK 17) • Emulator/Simulator setup (Android SDK, xcode) API & API Workflow tests: karate Contract tests: Specmatic Unit tests, Sonar code quality checks Test automation toolset Many environments Test data Branch Configuring appropriate test execution in build and release pipelines Complex path to production ADO agents: Windows Server & Linux agents Firewall restrictions download dependencies Direct access prohibited to CI agents Multiple node & JDK versions Connectivity issues to application-under-test No browsers/devices on CI agents CI execution @BagmarAnand
- 4.
- 5. Getting a simpleautomated test to run consistently for all Developers and Testers and in CI (ADO) is painful! Setup Execution (full or specific tests) @BagmarAnand
- 6. Challenges of End-2-End Test Automation Ensuring Test EnvironmentConsistency Coordinated Test Execution Test setup & execution on CI Agents @BagmarAnand
- 7.
- 8.
- 9. • Setup importantapplications on Mac • https://gist.github.com/anandbagmar/92b9f92298b1e17fa32c3404ad115 871 • Script to setup Android SDK on Mac • https://github.com/anandbagmar/AppiumJavaSample/blob/master/setup AndroidSDK.sh • Script to setup Android SDK on Linux • https://github.com/anandbagmar/AppiumJavaSample/blob/master/setup _linux.sh Test Authoring Environment Setup @BagmarAnand
- 10. • Node scriptto install all dependencies (for system tests): • https://github.com/znsio/getting-started-with- teswiz/blob/main/package.json • npm install – and you are ready! Test Execution Environment Setup @BagmarAnand
- 11.
- 12. • Setup shouldbe simple – Ex: • git pull • ./gradlew build • No code change required for • Running tests against any environment (local, dev, qa, staging, prod, etc.) • Test data and environment configurations are separately maintained • Running all or subset of tests • Tests should run from command-line Test Automation Framework Criteria @BagmarAnand
- 13. Your Ultimate Open-SourceSolution to Automate Real-User Scenarios! @BagmarAnand
- 14.
- 15. Run tests from CLI TestAuthoring Execution Setup 1 2 2 3 4 5 6 6 Execution Reports CI Tool Feature coverage @BagmarAnand 16
- 16. • Web browsers •Mobile-web browsers • Android apps • iOS apps • Windows desktop apps • Electron apps Platform support @BagmarAnand
- 17. • Open sourceframework to automate real-user scenarios • Multi-user • Multi-device • Multi-app • Setup a HARD-GATE for your functional tests! Unique capabilities of teswiz @BagmarAnand
- 18. • Cloud devicefarm integrations • Applitools AI for validations • Comprehensive reports with trend analysis, feature coverage, failure analysis using AI-ML • CLI • Configurable Unique capabilities of teswiz Defaults Property files Environment Variables @BagmarAnand
- 19.
- 20.
- 21. Use the rightnode version @BagmarAnand
- 22. Use the rightnode version @BagmarAnand
- 23.
- 24. • For SystemTests, the artifact (apk/app) could have been generated from another pipeline • This artifact needs to be available in local or cloud device before tests can start execution Script for downloading artifacts – Why? @BagmarAnand
- 25. • Understand theCI tool APIs • Script downloads the android/iOS artifact for: • Specific branch • Latest successful build, or a specific build number Script for downloading artifacts @BagmarAnand
- 26. • Script uploadsthe android/iOS artifact to your device farm OR • Teswiz can upload it automatically for you Script for downloading artifacts – Bonus! @BagmarAnand
- 27.
- 28. • Understand whatdependencies in your framework need proxy information. Ex: • Gradle/maven • Downloading newer version of browser drivers • Any external connectivity • Framework should be configurable to pass this at test execution time. Ex: • No proxy required from local laptop execution • Proxy required when running tests from CI Proxy Handling @BagmarAnand
- 29.
- 30.
- 31. Downloading dependencies - Uberjar #6 @BagmarAnand
- 32. • To reducethe number of dependencies to be downloaded, teswiz is built as a uber jar. • Specify only “teswiz” as a dependency in your test framework Uber jar @BagmarAnand
- 33. • Run asa java process • ./gradlew run Uber jar @BagmarAnand
- 34.
- 35. • CI agentsmay not have browsers installed • The installed browser may be an older version Run browser in docker – Why? @BagmarAnand
- 36. • Should allowsupport for any os/architecture • Should allow choosing the browser (ex: firefox, chrome, etc.) • Should allow starting the containers with specific project names and dynamic ports to prevent conflict with multiple test executions • Support specifying proxy information • Can be used on local laptops as well as in CI executions Run browser in docker @BagmarAnand https://github.com/znsio/teswiz/blob/main/dockerContainers.sh https://github.com/znsio/teswiz/blob/main/docker-compose-v3.yml
- 37. Template for running testsin build pipeline #8 @BagmarAnand
- 38.
- 39. • Create templates •Reuse with appropriate configuration parameters Running tests in Build Pipeline @BagmarAnand
- 40.
- 41. Task Groups forrunning tests in release pipeline #9 @BagmarAnand
- 42.
- 43. • Create TaskGroups • Include in each relevant stage of Release pipeline Running tests in Release Pipeline @BagmarAnand
- 44.
- 45.
- 46.
- 47.
- 48.
- 49.
- 50. Hard Gate Make yourtests valuable! #10 @BagmarAnand
- 51. @BagmarAnand What is aHard Gate! Why is it required? https://github.com/znsio/teswiz/blob/main/docs/HardGate.md • Automated tests should allow you to take decisions on product quality
- 52. @BagmarAnand What is aHard Gate! Why is it required? https://github.com/znsio/teswiz/blob/main/docs/HardGate.md • For every test execution cycle: • Passing tests are expected to pass • Known Failing tests are supposed to fail, unless: • The product (bug) is fixed, OR • The test is fixed/updated If either criteria is not met, the build should fail!
- 53. @BagmarAnand Hard Gate -Make your tests valuable! https://github.com/znsio/teswiz/blob/main/docs/HardGate.md Build passes if Hard Gate criteria is met. Build fails if • one or more passing tests have failed, or, • one or more failing tests have passed
- 54. @BagmarAnand Functional (e2e) Automationas Hard Gate! https://github.com/znsio/teswiz/blob/main/docs/HardGate.md
- 55. Applitools Visual AI forvalidations #11 @BagmarAnand
- 56.
- 57. 90% less codeto write & maintain with infinite coverage. WITH APPLITOOLS AI Every Element Is Validated They Look & Work Perfect Use Applitools Ultrafast Grid (UFG) - Test is simpler – one call to Applitools (eyes.checkWindow()) validates the full screen - Run the test once - Get results from all browsers automatically - Less test data - No additional load on the application environment @BagmarAnand
- 58. • Works forall platforms • Native & hybrid apps – android, iOS • Web browsers • Desktop applications • Electron applications • Seamless scaling using Applitools Ultrafast Grid @BagmarAnand Applitools Visual AI
- 59. Specify as manybrowsers with viewports and devices as required for validation You do not need to do cross-browser validation at the end anymore! @BagmarAnand AI-powered Cross Browser Test Automation
- 60. reportportal as a Centralreporting server #12 @BagmarAnand
- 61.
- 62. A central reportingserver for your organization @BagmarAnand
- 63. Test Execution Real-time status •See progress of launches currently in progress • Can also see details of tests that are currently running, till the point of execution @BagmarAnand
- 64. Test Execution Details– Device farm report link & Device logs • The link to the device farm test execution dashboard is available in the result • teswiz attaches browser logs/device logs automatically to the result in ReportPortal @BagmarAnand
- 65. Test Execution Details– with screenshots • The test result includes screenshots as captured by the test @BagmarAnand
- 66. Test Execution Details– Applitools Visual AI Validation Results • The test result includes the status of Applitools Visual AI validation • Link to the Applitools dashboard is available in the result @BagmarAnand
- 67. Test Execution TrendAnalysis • Each test shows the trend of its execution – giving an indication of (in)stability @BagmarAnand
- 68. Test Results –Next Steps • On investigation of the failed tests, mark the failures with appropriate reasons (as configured) @BagmarAnand
- 69. Auto-analysis of failedtests Analyse the failure reasons by Auto-Analyzer based on Machine Learning @BagmarAnand
- 70. Auto Analysis ofTest Failures • Why waste time marking the test failed for the same reason as last time? • ReportPortal can do this automatically for you with the Auto Analysis and Pattern Analysis feature @BagmarAnand
- 71. Auto Analysis ofTest Failures @BagmarAnand
- 72. Test Result Visualization Configure simpleand understandable reports • Create as many dashboards as relevant for the team • Dashboards may be for different persona/role, giving appropriate information @BagmarAnand
- 73. • Teswiz andkarate test frameworks can automatically upload test results to your reportportal server • sendToReportPortal: • https://github.com/znsio/sendToReportPortal/blob/main/importRes ultsAndUpdateAttributes.sh • Can upload junit test results generated by any type of tests to reportportal with relevant test execution metadata reportportal.io @BagmarAnand
- 74. Challenges • Ensuring Test EnvironmentConsistency • Coordinated Test Execution • Test setup & execution on CI Agents Solutions Consistent environment setup Test Automation Framework support Node setup Script for downloading artifacts Proxy Handling Downloading dependencies - Uber Jar Browsers in docker Template for build pipelines Task groups for release pipelines Hard Gate AI for validations Central reporting server Summary @BagmarAnand
- 75.