Safety is not our first priority

I was on a flight back to Australia when something in the routine safety demonstration video struck me with more significance than it had in the past.

"Safety is our first priority"

I began to realise just how often I had seen this value being espoused and how often I’ve come to take it for granted. The infrastructure and services I rely on every day put my safety as their top priority. Above making money. Above being innovative. Above customer service. If that organisation did not keep me safe whilst my livelihood was in their hands, the organisation would consider itself to have failed at its primary objective.

If you look around, you’ll find that this is an incredibly common organisational value. You will find it in any organisation that operates heavy machinery, any organisation that is responsible for taking care of a dangerous place and any organisation that is responsible for building infrastructure upon which the public relies. It is therefore no surprise that this value has been reflected in the customary law of negligence for centuries. When someone is vulnerable to your actions making them unsafe, you are under a duty of care to ensure that above all else, they are safe.

It is customary for all engineers in the United States and Canada to receive an iron ring upon their qualification and admission into the profession. It is folklore that the iron is sourced from the remains of the Quebec Bridge, which collapsed during construction in 1907, killing seventy five construction workers. The reason the bridge collapsed was that the engineers responsible for its design and construction cut corners to get the project to completion more quickly. Those workers were relying on the engineers to keep them safe and the engineers put profit ahead of the safety of those people. The ring is intended to be a permanent reminder to all of those in the profession that such errors should never be repeated.

The priority of safety in software

You will very rarely find this value in organisations that develop software. Most organisations are even allergic to it, including verbose disclaimers in software licences purporting to waive any implied warranty that the code you run is fit for purpose and won’t cause you trouble. Most licenses will require users to agree to numerous waivers and consent to all sorts of things which may be harmful to them. There is no way other professions would get away with that without raising an eyebrow. But for software, we don’t seem to care.

THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

This is puzzling, because at the same time, professional bodies representing the computer software industry purport to put the public interest as their number one priority. For instance, the Australian Computer Society’s code of professional ethics says that “You will place the interests of the public above those of personal, business or sectional interests.” This disconnect is disturbing when we think about the consequences that flow from such ignorance. A failure to have a safety oriented culture means that as so-called “engineers” our profession becomes complacent about the impact of our strategic and engineering choices upon users. It results in culture that puts the need to stay ahead and continue making a profit above the need to protect the legitimate safety interests of users. It leads to a discourse which implicitly accepts that failure is inevitable and can only be mitigated as opposed to prevented by making trade-offs which may reduce competitiveness but protect the public.

A resort to realism

Many computer software professionals would retort at this point delivering perfect quality software is just not realistic. The project may be on a tight budget or operating under a tight timeframe. It would not be possible to remain competitive and maintain such a high standard. And nevertheless, software professionals say, users have come to understand this reality as well. The software industry has remained unregulated since its inception and the benefits for society have been explosive growth of technology and innovation at a pace never before seen.

Such explosive growth and innovation has had real winners and losers. Over the last forty years technology has moved beyond the realm of academia and military research and become democratised. Computing is readily accessible in the age of the smartphone. It is also no surprise then that the wealth generated by computer software has become concentrated in the hands of the few who were best able to execute and facilitate this explosive growth.

But there are losers. It is becoming increasingly difficult, and now almost impossible for people to opt out of relying on software in their daily lives, just like it is now almost impossible to opt out of roads, electricity and plumbing. Software controls an increasingly higher degree of the infrastructure that we rely on every day and increasingly controls the way that we interact with each other, do business and access essential services. In an age where software grows explosively and operates the world, people have become increasingly vulnerable to the choices software development organisations make. Every day we hear of a new data leak. Confidential information is stolen and sold to the highest bidder. Lives are meddled with and lives are ruined. User facing software upon which people rely to make a livelihood breaks and those users shoulder the burden of the lost time or inconvenience. In extreme cases people die. Virtual bridges are collapsing around us and the best we can say as a profession is that such collapses are a cost society is willing to accept in the name of exponential progress.

Safety, is not our first priority.

What if safety were our first priority?

Engineering failures threatening the safety of others can be avoided so long as we are willing, as a profession, to make certain trade-offs. If you ever wonder why the pace of change for aviation, rail transportation and civil architecture seems to be glacial in comparison, it is because safety is the first priority.

If we want to make safety the first priority, there are some uncomfortable counter-realities that we may need to accept. It is plausible that in such an alternate timeline, the smartphone may have never been brought to the mass market, It is possible that social networking and the writeable web would have been considered and then quietly abandoned, due to the inherent risk associated with storing confidential information about millions of people in a central repository. Your operating system and applications may be considerably more stable, but the redundancy required to avoid takings risks would mean that its performance and scope of functionality would be heavily reduced.

These costs, are, I suspect, too much for our cutting-edge society and cut- throat business culture to bear. One thing I do believe though is that we can still put the safety of users as our number one priority by focusing on the 20% which will avoid 80% of the risk. We can, as a profession, still ask the hard questions which might affect the bottom line. We can still ask whether a particular feature, though innovative, might bring with it risks to the people who depend on us that those people would find unacceptable, even if it meant sacrificing the feature. We can still look at the systems we are designing and ask “do I understand the impact upon a person if this fails?” We can still sit down and make sure the math checks out before proceeding on a course of conduct which may have irreversible consequences.

If we just sat down and accepted that certain trade-offs will have to be made, we can be a profession that can be proud to say that safety is our first priority.

Thanks to Wayne Spilsbury for providing feedback on and editing this essay