Team Guide to Software Operability

Team Guides for Software

Foreword

Introduction

1. What is software operability and why should we care?
2. Where can operability techniques be used?
3. How to use this book
4. What is covered in this book
5. Why we wrote this book
6. Feedback and suggestions

1.What does good operability look like?

1. Key points
2. 1.1Use operability checklists to assess core operability
3. 1.2Assess operability with real people regularly
4. 1.3Provide a good User Experience for all agents and all users, external and internal
5. 1.4Treat operational aspects as product features: viable, configurable, deployable, diagnosable, reliable, well-performing, securable, observable, recoverable
6. 1.5Good operability does not necessarily mean good safety or ethics
7. 1.6Summary

2.Core practices for good software operability

1. Key points
2. 2.1Logging and metrics are the first features to implement
3. 2.2Enumerate all likely failure modes of the software
4. 2.3Use well-defined, meaningful event identifiers
5. 2.4Define and track at least one KPI or SLI per service
6. 2.5Include operational hooks as first-class features
7. 2.6‘DONE’ means working correctly in Production
8. 2.7Treat Operations as a high-skill activity
9. 2.8The software development team writes a draft Run Book
10. 2.9Avoid a separate ‘Production-ization’ or ‘Hardening’ phase
11. 2.10Avoid Production-specific tools
12. 2.11Talk about ‘operational features’, not ‘non-functional requirements’
13. 2.12Developers and Product Owners should be on-call
14. 2.13Make operational problems visible
15. 2.14Test for operability in a deployment pipeline
16. 2.15Summary

3.Use Run Book collaboration to increase operability and prevent operational issues

1. Key points
2. 3.1Operational aspects are very similar across many software systems
3. 3.2Use a Run Book template as a common baseline for operational aspects
4. 3.3Use a Run Book Dialogue Sheet to facilitate discovery and avoid ‘documentation fallacy’
5. 3.4Assess operability on a regular basis: every sprint, iteration, or week
6. 3.5Summary

4.Use modern log aggregation and metrics for deep operational insights

1. Key points
2. 4.1Use logging to help design and understand distributed systems
3. 4.2Collect and aggregate logs and metrics centrally using standard tools & software
4. 4.3Focus on collaboration, design decisions, and team experience
5. The power of log aggregation
6. 4.4Identify 2 or 3 key application metrics and test these early on
7. 4.5Run log aggregation and metrics locally on development workstations
8. 4.6Hide sensitive information at the point of logging
9. 4.7Use Structured Logging for greater meaning
10. 4.8Use Event IDs for visibility of application behaviour
11. 4.9Collaborate on Event IDs to enhance operability
12. 4.10Test your logging and metrics
13. 4.11Trace operations across system boundaries with correlation IDs
14. 4.12Adapt your logging and metrics techniques to the technology characteristics
15. 4.13Summary

5.Use well-defined readiness checks to increase operational confidence

1. 5.1Introduction
2. 5.2Define readiness checks so we know when a service is ‘ready’
3. 5.3Use Deployment Verification Tests (DVTs) to increase confidence in infrastructure
4. 5.4Expose Endpoint Healthchecks for persistent services to detect problems early
5. 5.5Provide custom diagnostic hooks to expose additional operational information
6. 5.6Run operational checks within a deployment pipeline to gain rapid feedback
7. Key points
8. 5.7Define a set of Service Readiness criteria to establish operational viability
9. 5.8Summary

6.Use information radiators and dashboards to drive effective behaviour and good psychological responses

1. 6.1Introduction
2. 6.2Invest time an effort in good dashboard design and information radiators
3. 6.3Avoid information overload - be selective about information on screen
4. 6.4Consider common psychological responses
5. 6.5Use dashboards to promote inter-team collaboration
6. 6.6Example: Using Dashboard visualisation in Formula 1
7. 6.7Be aware of typical mistakes with dashboard design
8. 6.8Summary

7.Make operability part of the software product

1. Key points
2. 7.1Overview: why we need a focus on operability
3. 7.2Use rich, time-series logging and metrics to drive product decisions
4. The power of metrics on dashboards
5. 7.3Go beyond the Agile “User Story” to address operability as a first-class concern
6. 7.4Use secondary User Personas to address operability aspects
7. 7.5Use a single backlog for visible features and operational features
8. 7.6Make operational aspects part of the team’s regular work
9. 7.7Address operational aspects from the very start and then throughout the delivery phase
10. 7.8Raise an alert if a team is spending less than ~30% of their time / effort / budget on operational aspects
11. 7.9Product Owners should be responsible for the operational success of the software
12. 7.10Developers, Testers, and Product Owners should be “on call” for operational problems
13. 7.11Understand the business case for operability
14. 7.12Encourage a culture of operability
15. 7.13Summary (?)

Appendix

1. Adapt your logging techniques to the technology characteristics
2. Understand how the complexity of modern distributed systems drives a need for a focus on operability

Terminology

References and further reading

1. Introduction
2. Chapter 1 - What does good operability look like?
3. Chapter 2 - Core Operability Practices
4. Chapter 3 - Use Run Book collaboration to increase operability and prevent operational issues
5. Chapter 4 - Use modern log aggregation for deep operational and insights
6. Chapter 5 - Use Deployment Verification Tests and Endpoint Healthchecks for rapid feedback on environments
7. Chapter 6 - Run operational checks within a deployment pipeline to gain rapid feedback and increased collaboration
8. Chapter 7 - Use information radiators and dashboards to drive effective behaviour and good psychological responses
9. Chapter 8 - Use operability as a differentiating aspect of your software
10. Appendix

Run Book template

1. Service or system overview
2. System characteristics
3. Required resources
4. Security and access control
5. System configuration
6. System backup and restore
7. Monitoring and alerting
8. Operational tasks
9. Maintenance tasks
10. Failover and Recovery procedures

Index

About the authors

1. Matthew Skelton
2. Alex Moore
3. Rob Thatcher

Conflux Books