I have seen it all. Issues accross all layers. Some examples:
- JQuery selector performance
- Nagle related drag
- The unoptimized SQL Query
- Physical infrastructure limitations
One or all of them together can impact application performance. In each of those cases, it is often the most obscure factors that cause the performance issues. A SQL expert once said, “Optimizing SQL is more of an art than a science.”
Another problem is our incomplete knowledge. No matter what you know about a technology, it is impossible to predict how it will perform up front. No doubt deep technical knowledge prevents serious flaws, but it never guarantees the performance question is answered.
A driver may have a flaw that only surfaces under specific network conditions created by your application. A developer might modify code on the master branch (to use Git lingo) late at night and not submit it for review. After the next build, the major flaw in the code causes days of production performance issues that overwhelm your support organization.
I heard a story about a team at Facebook that was struggling with some bizarre performance issues. They began digging into how the compiler was optimizing the emitted code and discovered that changing the order of variable declarations in their source resulted in a small, but measurable, performance improvement. Since Facebook is operating at such a huge scale, that small change allowed them to repurpose an entire rack of expensive server hardware. They can do this because they have teams of highly competent software engineers and computer scientists specifically tasked with solving these types of problems.
Many organizations simply cannot afford to invest in that level of expertise and effort. They might be able to create a high performing software development team, but due to the different levels of developer knowledge and discipline, verification often becomes an afterthought. This leads to rounds of failure and fixes. It is a painful way to work.
There are even those that say, “This is the way of things!” If this was always the way of things, software on airplanes wouldn’t be reliable. Bridges would be failing. No one would walk into a skyscraper. ByteStorm exists to promote the vision of a world where software quality is a priority for all organizations.
I know it can better. While networking with a fellow technical tester, he told me a true story that made me smile. He was on a team that invested up front in quality. They had a simple, yet robust, automation suite tied into a tight continuous integration process. On launch day, the entire team knew the technology was solid. They knew the launch would go well. It did. And the benefits to the company were astounding. The support costs were significantly lower than anticipated.
Just imagine that. It can be done. All it takes is focus and effort.