Uploaded bySalesforce Engineering
Quality Testing and Agile at Salesforce
Salesforce leverages agile methodologies to deliver three major releases annually, emphasizing collaboration between developers and quality engineers to maintain high quality standards. The company's approach includes practices like continuous planning, cross-functional teams, and a robust test management system to foster efficiency and innovation. Salesforce's growth has led to the creation of various programs to engage employees and encourage innovation while maintaining a focus on quality in their software development process.
EngineeringâŚ
More Related Content
Similar to Quality Testing and Agile at Salesforce
More from Salesforce Engineering
Quality Testing and Agile at Salesforce
- 1. Quality Testing andAgile at Salesforce Delivering three major releases a year Kelly Currier, Director Agile Delivery kcurrier@salesforce.com @kcducks /in/kcurrier Vladimir Gerasimov, Sr. Software Engineer, Quality vgerasimov@salesforce.com /in/vgerasimov87
- 2. Safe harbor statementunder the Private Securities Litigation Reform Act of 1995: This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward- looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services. The risks and uncertainties referred to above include â but are not limited to â risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site. Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements. Safe Harbor
- 3. Vladimir Gerasimov Sr. SoftwareEngineer, Quality Kelly Currier Director Agile Delivery
- 4. Our Goals Today: â˘Show you how Salesforce Technology Organization leverages Agile methodologies to deliver 3 major releases a year ⢠Explain the testing practices used to maintain the highest quality standards ⢠Show you how Developers and Testers collaborate ⢠Provide practical tips on how to start your own agile transformation today
- 5. In the EarlyDays⌠⢠Delivering consistently ⢠Easy with everyone in same office ⢠No hand-offs ⢠Clear communication
- 6. ⢠Incredible growth â˘Unpredictable releases ⢠Bottlenecks ⢠Lack of visibility ⢠Changing priorities Growing Pains CausedâŚ
- 7. 2000 2001 20022003 2004 2005 2006 Features Delivered per Team Days between Major Releases Time in between major releases increased
- 8.
- 9.
- 10. Respect people Eliminate waste Createknowledge Build quality in Deliver fast Optimize the whole Just in time decisions 7 Principles are the Foundation of our Agility
- 11. Vision V2MOM 2 Release Review 1Release Review Sprint Daily Continuous Planning & Prioritization at Every Level Spring Summer 3-5 Years 1 year R1 2 Releases 1 Release ~ 2 Weeks Daily Vision Values Method Obstacles Metrics
- 12. It all startswith V2MOM CEO Technology Organization Cloud CloudCloud Cloud Team Team Team TeamTeam Team Team Team Team Team Connected MeasurementIterative PrioritizationTotal Alignment Vision: What impact will you have this year? Values: What is most important to you? Method: Actions required to achieve vision? Obstacles: What might stand in your way? Measures: Results youâre aiming to achieve?
- 13. Roles within ourSoftware Engineering Organization Team ⢠Cross functional (5-9) (Dev, QE, Doc, UX, Perf, DevOps*) ⢠Self organizes around how to deliver ⢠Determines how much to take on Product Owner ⢠Defines project goals ⢠Gathers/prioritizes requirements ⢠Writes stories & acceptance criteria ⢠Makes scope/schedule trade-offs ⢠Provides continuous feedback during sprint Scrum Master/Kanban Master ⢠Enable self-organization ⢠Facilitate continuous improvement ⢠Coach team and stakeholders ⢠Remove obstacles ⢠Provide visibility Functional Manager ⢠Technical excellence & standards ⢠Hiring, people development ⢠Removing blockers ⢠Empowering team members Technical Program Manager ⢠Drives delivery for a program or cloud ⢠Visibly & transparently reports progress ⢠Aligns everyone to common goals
- 14. 2007 2008 20092010 2011 2012 2013 Features Delivered per Team Days between Major Releases Agile Enables 3 Predictable Major Releases a Year
- 15. 400 + Agile Teams@ Salesforce Weâve Built a Scalable Model
- 16. Innovation Programs RetainTop Talent Opportunity Open Market What: Opportunity for employees to easily move positions to another role, product, feature area. Goal: Retain top talent. Spread knowledge. Timing: 3/year PTOn! What: Paid time per month to innovate, explore, learn, hack and step out of the day to day. Goal: Allow employees to learn and innovate - factors that drive them and their success. Timing: Ongoing. Shark Tank What: Pitch an idea to a panel of âinvestorsâ. Goal: Inspire employees to take risks, work with others and provide time for innovation Timing: 3/year
- 17.
- 18. Quality Engineers atSalesforce are Responsible for: ⢠Test Design and Test Planning ⢠Identifying Test Coverage gaps ⢠Developing Test Frameworks ⢠Writing Test Automation
- 19. User Story: CollaborationBetween Devs and QEs New Triaged In Progress Ready For Review Fixed QE in Progress Closed ⢠User Story cannot be closed without QE ⢠Devs and QEs working on the stories together Devs and QEs discussed the work Devs and QEs reviewed the code Code checked in Definition of Done is met
- 20. Dog-Fooding our ownproduct Bug and User Story tracking Internal and External Communication Helps discover issues earlier Using your product on daily basis
- 21. Definition of Done â˘Single Definition of Done ⢠Establishes shared, explicit understanding within & across teams ⢠Increases team accountability for quality and commitment ⢠Reduces waste ⢠Encourages cross-team collaboration at multiple touch points Criteria Features 1 2 3 Code checked in and follows department standards â â No open regressions. Automated tests written and reviewed for all regressions â â No open P1 & P2 bugs â Code Coverage 78% 80% 100% of test cases logged and executed in a QA environment, and all P1/P2 cases passing â â All resolved bugs verified and closed â â â Performance/scalability impact ascertained and sys testing scheduled if required â â â UE has reviewed any new features; P1 and P2 UI bugs fixed â â â Usability testing completed when necessary, and feedback incorporated into backlog â â Code and UI reviewed for 508 compliance; UE team notified of any non-compliant features â â All UI labels ready for localization vendors â â User documentation complete and checked in â
- 22. Frequent Product Demos& Reviews ⢠Demo completed work with team every sprint ⢠Demo and review release plan progress monthly with Executive participation ⢠Provide radical transparency ⢠No surprises come release time
- 23. Freedom to adjustthe process Teams decide what work or doesnât work for them ⢠Team has the freedom to decide best practices ⢠Continuous improvement part of the DNA ⢠Encourages everyone to own the process ⢠Reduces amount of bureaucracy
- 24.
- 25. Test Failures, Flappersand Long Running Tests Lock teams with Test Failures older than 7 days Opening Bugs for non-stable tests Opening Bugs for Long Running Tests Keeping test healthy and stable
- 26. Lock Out Email Teamgets locked out if they don't fix their Test Failures
- 27. Test History Providing reportsto help analyze test failure Change List that most likely caused the failure Stacktrace information
- 28. Test Failure AutoAssign Auto-Assign manages test failures effectively It can link Test Failure to the change list that most likely caused the failure It groups similar test failures under one Bug It eliminates manual work of running reports and opening bugs Even when wrong, it still opens bugs and starts the process of fixing Test Failure Helping teams to address test failures
- 29.
- 30. Bug Triage Helps toassign bugs to engineers ⢠Usually happens on weekly basis ⢠Helps address Bugs that werenât auto-assigned ⢠Helps keep bug-log clean ⢠Devs and QEs attend
- 31. Pre-Checkin Validation Keeps theBuild healthy ⢠Sharing the same code branch for current release work ⢠Important to keep build healthy at all times ⢠Pre-checkin validation ensures the change you are submitting is not breaking the build and major functionality ⢠Takes about 1.5 hours
- 32. Frequent Major Releases Feature Freeze Release Freeze DoneDone Done Release to Internal Sandbox & Production Instances SB/R0 Release ⢠2/3 sandbox instances ⢠Production instance where Salesforce has largest orgs R1/R2 Release ⢠25% of prod instances ⢠All remaining instances ⢠Branch locked ⢠Check-in approval required ⢠Incomplete features disabled ⢠Code line open for next release Monthly Sprint Reviews Release Sprint Staggered Release Scrum Teams and Functional Areas Sign Off Scrum Teams Sign Off Dec Jan Feb Mar MayApr Jun ⢠Continuous integration w/ 600k JUnit and Selenium tests ⢠Performance testing ⢠110M Apex customer tests ⢠Other production tests ⢠Final performance testing
- 33. Next Steps What youcan do today Get Developers to work closer with Quality Engineers ⢠Discuss User Story scope and requirements together ⢠Participate in Code Reviews ⢠Have Dev and QE pair on story completion Test Automation ⢠Add Automated Test Coverage to Definition of Done ⢠Create backlog to track remaining Automation work ⢠Involve Developers in writing Automation and helping QEs with Test Frameworks ⢠Test Failures locks when too high ⢠Enforce Test Automation health ⢠Invest in Auto-Assign tools and reports
- 34.