Welcome!

@ThingsExpo Authors: Yeshim Deniz, Elizabeth White, Pat Romanski, Liz McMillan, Carmen Gonzalez

Related Topics: @ThingsExpo, Java IoT, Agile Computing, Wearables, @CloudExpo

@ThingsExpo: Blog Feed Post

You Can’t Test All the 'Things' | @ThingsExpo #IoT #M2M #API #InternetOfThings

Young engineers building all sorts of IoT devices without a thought to protocol or integration standards

You Can’t Test All the Things: API, IoT, ROI TBD

I have three words for everyone in software testing: prioritize, prioritize, and prioritize.

You can't test every possible permutation of your software, especially so with APIs and IoT devices where you're placing much of the user experience in the hands of integrators to your core products and services. You just can't, so we should throw our hands up now and just give up, right?

What sort of tester are you anyway?

I'm not; at best I'm an automation engineer. I don't nearly have the mind or patience that is required of a full-time, dedicated testing professional. But I talk to them and lots of other people every day, and I'm still a developer by heart, so I feel very kin to everyone involved in the software delivery process.

You don't have to be a tester to understand the constraints of being human. Humans make mistakes and only have so much time per day. Developers who are starting to experience a shift-left in testing responsibilities might make sure that all functions and procedures are unit tested, but often don't go much further simply because there isn't enough time. Not enough people, not enough time...two things I hear an awful lot and part of the reason why I'm in the automated testing tools business with Ready! API.

Side note: I would also like to take this opportunity to say that "no budget" and "not a priority" are the worst reasons not to invest in a better future, one where automation gives you back time to build better software and where getting your delivery pipeline right is critical to moving forward with rapid iteration.

But shouldn't we always test more?

No. Well, yes. Actually, sort of and not always. There's an aspect of diminishing return to introducing more testing in that if you don't prioritize the most business-critical components and workflows to be tested first, you fail your job, your business, and yourself as a professional tester.

We simply can test everything, we just don't have the time, and we also need to stay nimble. If I were a tester, I'd probably get fired for wanting to confirm that what I was told to work on actually affects the bottom line of the business in the most impactful way at this moment in time. I would not sit there creating tests just so that we could report that we have 79% instead of 78% testing coverage today. I would want to see ROI, even though apparently "there ain't no ROI in software testing".

What about all the new stuff, shouldn't that be tested?

I got a question after my talk at APIdays IoT San Francisco 2015 in July: "If you don't know how people are going to combine your devices and APIs, how can you ensure that you've tested everything?" In no disrespectful way, this kind of question underlines the immaturity of the IoT space. We've got a very young startup culture (think 20-somethings with multiple PhDs) attacking big problems, most usually around how the digital world imbues itself back into the physical world, as quickly as possible for as cheap as possible.

There's going to be some broken things, incomplete thoughts, and definitely failures to learn from. I saw this first hand at O'Reilly Solid the week after APIdays too. The speakers were experienced industry leaders, and it was good that those people were on stage speaking to the young engineers building all sorts of IoT devices without a thought to protocol or integration standards. The median booth sponsor was grad school geniuses that are practically DevOps natives building our future...so busy implementing that they may not even know to ask the more important question, "should we"?

What do you have against startups?

Nothing personal, I've been involved in a few myself. The "startup kids" are working incredibly hard, many harder than I've ever worked, to make it or break it in the cutthroat technology market that we've built for ourselves. And when I say "kids", I mean figuratively; folks who bring a variety of backgrounds, skills, and experience to a startup still often have a lot to learn about running a lean operation before things lock in and start whirring.

Some things take time to learn (thanks to our lizard brains) and we (the technology industry) would do well to remember that people are still naturally slow to accept and properly utilize new technologies (like for instance the slow but increasing industry adoption of lightweight service virtualization tools).

If there was one thing I'd take issue with related to startup culture, it's that some important corners can get cut when you're so busy accelerating that you don't have time to ask "should we?". For instance, API security is a huge deal now because of a number of very public API security failures have occurred in the past few years. Moonpig, Uber, Tinder, and others could have avoided this just by applying free knowledge to their designs.

The stakes get bigger too the closer to money and safety your software is, like as in financial and medical institutions, or worse defense and air travel; proactively identifying vulnerabilities to critical services is a fundamental reason for regulatory audits, and keep omissions in safety introduced by startup culture thinking from having a massive negative affect on everyday life for the average digital citizen.

Then how do I know when I've tested enough?

That's up to you, it's your software, it's your customers and their expectations that should help you know when enough is enough. In the absence of formal UX and operational metrics, you may need to set some goals starting with your technical team. You should also be talking directly with your consumers, getting their feedback and input on what they expect from your software.

Go read some James Bach or Michael Bolton books if you want to learn about testing in general, but when it comes down to specifics like SLAs and reporting to management, there are very tangible questions to ask about quality. For instance, a slightly more matured set of questions around testing IoT and other multiples of design complexity (like Microservices, Hypermedia, etc.) might include:

  • Should my goal be to test all the things and permutations?
  • What do the user-experience metrics say about what my customers see as a priority?
  • What technical omissions contribute to our worst business deficits?
  • Do I need to build/test/ship this, right now, at this very moment?
  • What do we gain and/or lose by shipping what we have, such as it is?

Okay, so I don't have to test more?

Don't we all wish; sorry, no, this isn't a free lunch. Yes, you should be testing more, almost universally. Most software vendors and engineers know that automated testing is the only way to achieve high quality with less manual intervention, but haven't seen the pain that a lack of proper testing causes.

Good engineers and technicians have consciences and know when something needs more attention, which is why I leave it up to you to answer the question "how much testing do I need?" You don't need your inner voice telling you that you need to test and monitor something that only you know would cripple your business if it went down for any amount of time. So in those moments, I humbly submit to you that if you see or feel the future pain, it is your responsibility to write the test, or at least put it in the priority backlog, or something that your team is sure to follow up on.

Often backlogged is "test coverage", a useful assessment tool, but also just a broad-spectrum approach to getting visibility over technical gaps in software delivery. It helps you see when new things are implemented without the proper team communication, but should not be a goal on its own right. When was the last time your customer said "if you don't have at least 82% testing coverage, I don't want to use your software"? Right. Most consumers care about the downstream effects of bad software like app crashes, not the things you do to prevent bugs from getting to production. Coverage is not a golden idol, it's just a method of assessing where your weaknesses lie.

Like security testing, test coverage is a measurement of only one aspect of readiness that should be part of a larger perspective on customer-focused software quality. The connection between testing and customer value shows up in other areas like beta acceptance testing, performance monitoring, and consumer feedback. It's there for the taking when we want it.

That's my experience with testing "all the things", but what's yours? I'd love to hear your thoughts.

Read the original blog entry...

More Stories By SmartBear Blog

As the leader in software quality tools for the connected world, SmartBear supports more than two million software professionals and over 25,000 organizations in 90 countries that use its products to build and deliver the world’s greatest applications. With today’s applications deploying on mobile, Web, desktop, Internet of Things (IoT) or even embedded computing platforms, the connected nature of these applications through public and private APIs presents a unique set of challenges for developers, testers and operations teams. SmartBear's software quality tools assist with code review, functional and load testing, API readiness as well as performance monitoring of these modern applications.

IoT & Smart Cities Stories
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...
Early Bird Registration Discount Expires on August 31, 2018 Conference Registration Link ▸ HERE. Pick from all 200 sessions in all 10 tracks, plus 22 Keynotes & General Sessions! Lunch is served two days. EXPIRES AUGUST 31, 2018. Ticket prices: ($1,295-Aug 31) ($1,495-Oct 31) ($1,995-Nov 12) ($2,500-Walk-in)
IoT is rapidly becoming mainstream as more and more investments are made into the platforms and technology. As this movement continues to expand and gain momentum it creates a massive wall of noise that can be difficult to sift through. Unfortunately, this inevitably makes IoT less approachable for people to get started with and can hamper efforts to integrate this key technology into your own portfolio. There are so many connected products already in place today with many hundreds more on the h...
Digital Transformation is much more than a buzzword. The radical shift to digital mechanisms for almost every process is evident across all industries and verticals. This is often especially true in financial services, where the legacy environment is many times unable to keep up with the rapidly shifting demands of the consumer. The constant pressure to provide complete, omnichannel delivery of customer-facing solutions to meet both regulatory and customer demands is putting enormous pressure on...
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
Machine learning has taken residence at our cities' cores and now we can finally have "smart cities." Cities are a collection of buildings made to provide the structure and safety necessary for people to function, create and survive. Buildings are a pool of ever-changing performance data from large automated systems such as heating and cooling to the people that live and work within them. Through machine learning, buildings can optimize performance, reduce costs, and improve occupant comfort by ...
Charles Araujo is an industry analyst, internationally recognized authority on the Digital Enterprise and author of The Quantum Age of IT: Why Everything You Know About IT is About to Change. As Principal Analyst with Intellyx, he writes, speaks and advises organizations on how to navigate through this time of disruption. He is also the founder of The Institute for Digital Transformation and a sought after keynote speaker. He has been a regular contributor to both InformationWeek and CIO Insight...
Digital Transformation: Preparing Cloud & IoT Security for the Age of Artificial Intelligence. As automation and artificial intelligence (AI) power solution development and delivery, many businesses need to build backend cloud capabilities. Well-poised organizations, marketing smart devices with AI and BlockChain capabilities prepare to refine compliance and regulatory capabilities in 2018. Volumes of health, financial, technical and privacy data, along with tightening compliance requirements by...
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
Nicolas Fierro is CEO of MIMIR Blockchain Solutions. He is a programmer, technologist, and operations dev who has worked with Ethereum and blockchain since 2014. His knowledge in blockchain dates to when he performed dev ops services to the Ethereum Foundation as one the privileged few developers to work with the original core team in Switzerland.