Download as PDF, PPTX
Uploaded byIlya Puchka
Testing sync engine
This document discusses testing strategies for the iOS Sync Engine. It provides overall test stats, including that unit tests make up 80% of tests and integration tests make up 20%. The main testing strategy is to test against a fake backend called MockTransportSession, which provides advantages like speed, flexibility and independence over using stubs. MockTransportSession simulates the real backend by making fake requests, managing an internal database, and responding to requests. Future plans include splitting tests into smaller frameworks to reduce runtime, running tests on pull requests, and using record/replay to keep MockTransportSession in sync with the real backend.
![[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...](https://cdn.slidesharecdn.com/ss_thumbnails/dpesummitseptember20-212023-howimprovingthetestingexperiencegoesbeyondqualityadeveloperproductivityp-230924000523-453e8af8-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Testing sync engine
- 1.
- 2. Testing (iOS) SyncEngine
- 3. Agenda • Overall testsstats • Our test strategies • MockTransportSession • Workflow • Problems • Future plans
- 4.
- 5. • 3356 testson OSX Overall tests stats
- 6. • 3356 testson OSX • 3415 tests on iOS (+ apns tests) Overall tests stats
- 7. • 3356 testson OSX • 3415 tests on iOS (+ apns tests) • ~ 7.5 minutes to build and run all the tests on one platform (build ~ 40s, tests ~ 7 min) Overall tests stats
- 8. • 3356 testson OSX • 3415 tests on iOS (+ apns tests) • ~ 7.5 minutes to build and run all the tests on one platform (build ~ 40s, tests ~ 7 min) Overall tests stats
- 9. • Unit tests~ 80% Overall tests stats
- 10. • Unit tests~ 80% • Integration tests ~ 20% Overall tests stats
- 11. • Unit tests~ 80% • Integration tests ~ 20% • Performance tests - 18 Overall tests stats
- 12. • Unit tests~ 80% • Integration tests ~ 20% • Performance tests - 18 • Tests of tests Overall tests stats
- 13. • Unit tests~ 80% • Integration tests ~ 20% • Performance tests - 18 • Tests of tests • Sorry, no UI tests Overall tests stats
- 14.
- 15. What we DON’Tdo: Our test strategies
- 16. What we DON’Tdo: • test against http:// (real backend) • test using http requests stubs (like OHHTTPStubs) Our test strategies
- 17. What we DON’Tdo: • test against http:// (real backend) • test using http requests stubs (like OHHTTPStubs) Why? • Stubs are not that flexible. Our test strategies
- 18. What we DON’Tdo: • test against http:// (real backend) • test using http requests stubs (like OHHTTPStubs) Why? • Stubs are not that flexible. • It’s still slow. Our test strategies
- 19. What we reallydo: • Test against fake backend Our test strategies
- 20. What we reallydo: • Test against fake backend Advantages • Fast • Flexible • Independent • Testable Our test strategies
- 21. What we reallydo: • Test against fake backend Advantages • Fast • Flexible • Independent • Testable Disadvantages • Need to write (and test!) code • Need to be in sync with real backend (can be improved) Our test strategies
- 22.
- 23.
- 24.
- 25.
- 26. WTF is THAT? ZMTransportSession- is a delegate of a delegate of NSURLSession MockTransportSession
- 27. WTF is THAT? ZMTransportSession- is a delegate of a delegate of NSURLSession • Sends http requests to scheduler MockTransportSession
- 28. WTF is THAT? ZMTransportSession- is a delegate of a delegate of NSURLSession • Sends http requests to scheduler • Handles authentication MockTransportSession
- 29. WTF is THAT? ZMTransportSession- is a delegate of a delegate of NSURLSession • Sends http requests to scheduler • Handles authentication • Opens/closes web socket connection MockTransportSession
- 30. WTF is THAT? MockTransportSession- is a mock of ZMTransportSession MockTransportSession
- 31. WTF is THAT? MockTransportSession- is a mock of ZMTransportSession MockTransportSession - acts as a fake backend MockTransportSession
- 32. WTF is THAT? •Make fake requests MockTransportSession
- 33. WTF is THAT? •Make fake requests • Records requests and responds to them MockTransportSession
- 34. WTF is THAT? •Make fake requests • Records requests and responds to them • Manages internal database MockTransportSession
- 35. WTF is THAT? •Make fake requests • Records requests and responds to them • Manages internal database • Can simulate remote changes. MockTransportSession
- 36. WTF is THAT? •Make fake requests • Records requests and responds to them • Manages internal database • Can simulate remote changes. • Can send push channel (web socket) events MockTransportSession
- 37. • Before eachtest we insert objects in internal database (in memory) MockTransportSession
- 38. • Before eachtest we insert objects in internal database (in memory) • Test code make request using MockTransportSession MockTransportSession
- 39. • Before eachtest we insert objects in internal database (in memory) • Test code make request using MockTransportSession • It records and process incoming request (filtering by path and lots of ifs) MockTransportSession
- 40. • Before eachtest we insert objects in internal database (in memory) • Test code make request using MockTransportSession • It records and process incoming request (filtering by path and lots of ifs) • Performs some CRUD action on database depending on request (add message, create conversation, fetch conversation) MockTransportSession
- 41. • Before eachtest we insert objects in internal database (in memory) • Test code make request using MockTransportSession • It records and process incoming request (filtering by path and lots of ifs) • Performs some CRUD action on database depending on request (add message, create conversation, fetch conversation) • Creates and sends response. Optionally can delegate it to other object (i.e. test case). MockTransportSession
- 42.
- 43. • Finished feature- run all the tests Workflow
- 44. • Finished feature- run all the tests • Create pull request Workflow
- 45. • Finished feature- run all the tests • Create pull request • Review, merge Workflow
- 46. • Finished feature- run all the tests • Create pull request • Review, merge • Release script runs all tests locally Workflow
- 47. • Finished feature- run all the tests • Create pull request • Review, merge • Release script runs all tests locally • Pushes to remote branches (master/develop) Workflow
- 48. • Finished feature- run all the tests • Create pull request • Review, merge • Release script runs all tests locally • Pushes to remote branches (master/develop) • Xcode bots make integration on each push to master/ develop Workflow
- 49. • Finished feature- run all the tests • Create pull request • Review, merge • Release script runs all tests locally • Pushes to remote branches (master/develop) • Xcode bots make integration on each push to master/ develop • Each bot for each platform runs all the tests Workflow
- 50. • Finished feature- run all the tests • Create pull request • Review, merge • Release script runs all tests locally • Pushes to remote branches (master/develop) • Xcode bots make integration on each push to master/ develop • Each bot for each platform runs all the tests • Bots archive and upload binary to the cloud Workflow
- 51.
- 52. Solved: • Tests isolation(mostly database state) Problems
- 53. Solved: • Tests isolation(mostly database state) • Handling asynchronous code Problems
- 54. Solved: • Tests isolation(mostly database state) • Handling asynchronous code Not yet solved: Problems
- 55. Solved: • Tests isolation(mostly database state) • Handling asynchronous code Not yet solved: • Reduce tests running time U don’t like me? =( Problems
- 56. Solved: • Tests isolation(mostly database state) • Handling asynchronous code Not yet solved: • Reduce tests running time • Sync with real backend Problems
- 57. Solved: • Tests isolation(mostly database state) • Handling asynchronous code Not yet solved: • Reduce tests running time • Sync with real backend • Flaky tests Problems
- 58.
- 59. • Split tosmaller frameworks - to improve run time Future plans
- 60. • Split tosmaller frameworks - to improve run time • Setup Buildasaur - to run tests on Github pull requests Future plans
- 61. • Split tosmaller frameworks - to improve run time • Setup Buildasaur - to run tests on Github pull requests • Record/replay tests for MockTransportSession - to keep it in sync Future plans
- 62.