Welcome!

@ThingsExpo Authors: Yeshim Deniz, Elizabeth White, Liz McMillan, Karyn Jeffery, William Schmarzo

Related Topics: @ThingsExpo, Java IoT, Agile Computing, Wearables, @CloudExpo

@ThingsExpo: Blog Feed Post

You Can’t Test All the 'Things' | @ThingsExpo #IoT #M2M #API #InternetOfThings

Young engineers building all sorts of IoT devices without a thought to protocol or integration standards

You Can’t Test All the Things: API, IoT, ROI TBD

I have three words for everyone in software testing: prioritize, prioritize, and prioritize.

You can't test every possible permutation of your software, especially so with APIs and IoT devices where you're placing much of the user experience in the hands of integrators to your core products and services. You just can't, so we should throw our hands up now and just give up, right?

What sort of tester are you anyway?

I'm not; at best I'm an automation engineer. I don't nearly have the mind or patience that is required of a full-time, dedicated testing professional. But I talk to them and lots of other people every day, and I'm still a developer by heart, so I feel very kin to everyone involved in the software delivery process.

You don't have to be a tester to understand the constraints of being human. Humans make mistakes and only have so much time per day. Developers who are starting to experience a shift-left in testing responsibilities might make sure that all functions and procedures are unit tested, but often don't go much further simply because there isn't enough time. Not enough people, not enough time...two things I hear an awful lot and part of the reason why I'm in the automated testing tools business with Ready! API.

Side note: I would also like to take this opportunity to say that "no budget" and "not a priority" are the worst reasons not to invest in a better future, one where automation gives you back time to build better software and where getting your delivery pipeline right is critical to moving forward with rapid iteration.

But shouldn't we always test more?

No. Well, yes. Actually, sort of and not always. There's an aspect of diminishing return to introducing more testing in that if you don't prioritize the most business-critical components and workflows to be tested first, you fail your job, your business, and yourself as a professional tester.

We simply can test everything, we just don't have the time, and we also need to stay nimble. If I were a tester, I'd probably get fired for wanting to confirm that what I was told to work on actually affects the bottom line of the business in the most impactful way at this moment in time. I would not sit there creating tests just so that we could report that we have 79% instead of 78% testing coverage today. I would want to see ROI, even though apparently "there ain't no ROI in software testing".

What about all the new stuff, shouldn't that be tested?

I got a question after my talk at APIdays IoT San Francisco 2015 in July: "If you don't know how people are going to combine your devices and APIs, how can you ensure that you've tested everything?" In no disrespectful way, this kind of question underlines the immaturity of the IoT space. We've got a very young startup culture (think 20-somethings with multiple PhDs) attacking big problems, most usually around how the digital world imbues itself back into the physical world, as quickly as possible for as cheap as possible.

There's going to be some broken things, incomplete thoughts, and definitely failures to learn from. I saw this first hand at O'Reilly Solid the week after APIdays too. The speakers were experienced industry leaders, and it was good that those people were on stage speaking to the young engineers building all sorts of IoT devices without a thought to protocol or integration standards. The median booth sponsor was grad school geniuses that are practically DevOps natives building our future...so busy implementing that they may not even know to ask the more important question, "should we"?

What do you have against startups?

Nothing personal, I've been involved in a few myself. The "startup kids" are working incredibly hard, many harder than I've ever worked, to make it or break it in the cutthroat technology market that we've built for ourselves. And when I say "kids", I mean figuratively; folks who bring a variety of backgrounds, skills, and experience to a startup still often have a lot to learn about running a lean operation before things lock in and start whirring.

Some things take time to learn (thanks to our lizard brains) and we (the technology industry) would do well to remember that people are still naturally slow to accept and properly utilize new technologies (like for instance the slow but increasing industry adoption of lightweight service virtualization tools).

If there was one thing I'd take issue with related to startup culture, it's that some important corners can get cut when you're so busy accelerating that you don't have time to ask "should we?". For instance, API security is a huge deal now because of a number of very public API security failures have occurred in the past few years. Moonpig, Uber, Tinder, and others could have avoided this just by applying free knowledge to their designs.

The stakes get bigger too the closer to money and safety your software is, like as in financial and medical institutions, or worse defense and air travel; proactively identifying vulnerabilities to critical services is a fundamental reason for regulatory audits, and keep omissions in safety introduced by startup culture thinking from having a massive negative affect on everyday life for the average digital citizen.

Then how do I know when I've tested enough?

That's up to you, it's your software, it's your customers and their expectations that should help you know when enough is enough. In the absence of formal UX and operational metrics, you may need to set some goals starting with your technical team. You should also be talking directly with your consumers, getting their feedback and input on what they expect from your software.

Go read some James Bach or Michael Bolton books if you want to learn about testing in general, but when it comes down to specifics like SLAs and reporting to management, there are very tangible questions to ask about quality. For instance, a slightly more matured set of questions around testing IoT and other multiples of design complexity (like Microservices, Hypermedia, etc.) might include:

  • Should my goal be to test all the things and permutations?
  • What do the user-experience metrics say about what my customers see as a priority?
  • What technical omissions contribute to our worst business deficits?
  • Do I need to build/test/ship this, right now, at this very moment?
  • What do we gain and/or lose by shipping what we have, such as it is?

Okay, so I don't have to test more?

Don't we all wish; sorry, no, this isn't a free lunch. Yes, you should be testing more, almost universally. Most software vendors and engineers know that automated testing is the only way to achieve high quality with less manual intervention, but haven't seen the pain that a lack of proper testing causes.

Good engineers and technicians have consciences and know when something needs more attention, which is why I leave it up to you to answer the question "how much testing do I need?" You don't need your inner voice telling you that you need to test and monitor something that only you know would cripple your business if it went down for any amount of time. So in those moments, I humbly submit to you that if you see or feel the future pain, it is your responsibility to write the test, or at least put it in the priority backlog, or something that your team is sure to follow up on.

Often backlogged is "test coverage", a useful assessment tool, but also just a broad-spectrum approach to getting visibility over technical gaps in software delivery. It helps you see when new things are implemented without the proper team communication, but should not be a goal on its own right. When was the last time your customer said "if you don't have at least 82% testing coverage, I don't want to use your software"? Right. Most consumers care about the downstream effects of bad software like app crashes, not the things you do to prevent bugs from getting to production. Coverage is not a golden idol, it's just a method of assessing where your weaknesses lie.

Like security testing, test coverage is a measurement of only one aspect of readiness that should be part of a larger perspective on customer-focused software quality. The connection between testing and customer value shows up in other areas like beta acceptance testing, performance monitoring, and consumer feedback. It's there for the taking when we want it.

That's my experience with testing "all the things", but what's yours? I'd love to hear your thoughts.

Read the original blog entry...

More Stories By SmartBear Blog

As the leader in software quality tools for the connected world, SmartBear supports more than two million software professionals and over 25,000 organizations in 90 countries that use its products to build and deliver the world’s greatest applications. With today’s applications deploying on mobile, Web, desktop, Internet of Things (IoT) or even embedded computing platforms, the connected nature of these applications through public and private APIs presents a unique set of challenges for developers, testers and operations teams. SmartBear's software quality tools assist with code review, functional and load testing, API readiness as well as performance monitoring of these modern applications.

@ThingsExpo Stories
SYS-CON Events announced today that Calligo, an innovative cloud service provider offering mid-sized companies the highest levels of data privacy and security, has been named "Bronze Sponsor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Calligo offers unparalleled application performance guarantees, commercial flexibility and a personalised support service from its globally located cloud plat...
IoT is at the core or many Digital Transformation initiatives with the goal of re-inventing a company's business model. We all agree that collecting relevant IoT data will result in massive amounts of data needing to be stored. However, with the rapid development of IoT devices and ongoing business model transformation, we are not able to predict the volume and growth of IoT data. And with the lack of IoT history, traditional methods of IT and infrastructure planning based on the past do not app...
With tough new regulations coming to Europe on data privacy in May 2018, Calligo will explain why in reality the effect is global and transforms how you consider critical data. EU GDPR fundamentally rewrites the rules for cloud, Big Data and IoT. In his session at 21st Cloud Expo, Adam Ryan, Vice President and General Manager EMEA at Calligo, will examine the regulations and provide insight on how it affects technology, challenges the established rules and will usher in new levels of diligence...
SYS-CON Events announced today that DXWorldExpo has been named “Global Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Digital Transformation is the key issue driving the global enterprise IT business. Digital Transformation is most prominent among Global 2000 enterprises and government institutions.
SYS-CON Events announced today that Datera, that offers a radically new data management architecture, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Datera is transforming the traditional datacenter model through modern cloud simplicity. The technology industry is at another major inflection point. The rise of mobile, the Internet of Things, data storage and Big...
In his session at @ThingsExpo, Sudarshan Krishnamurthi, a Senior Manager, Business Strategy, at Cisco Systems, discussed how IT and operational technology (OT) work together, as opposed to being in separate siloes as once was traditional. Attendees learned how to fully leverage the power of IoT in their organization by bringing the two sides together and bridging the communication gap. He also looked at what good leadership must entail in order to accomplish this, and how IT managers can be the ...
DX World EXPO, LLC., a Lighthouse Point, Florida-based startup trade show producer and the creator of "DXWorldEXPO® - Digital Transformation Conference & Expo" has announced its executive management team. The team is headed by Levent Selamoglu, who has been named CEO. "Now is the time for a truly global DX event, to bring together the leading minds from the technology world in a conversation about Digital Transformation," he said in making the announcement.
Everything run by electricity will eventually be connected to the Internet. Get ahead of the Internet of Things revolution and join Akvelon expert and IoT industry leader, Sergey Grebnov, in his session at @ThingsExpo, for an educational dive into the world of managing your home, workplace and all the devices they contain with the power of machine-based AI and intelligent Bot services for a completely streamlined experience.
In the enterprise today, connected IoT devices are everywhere – both inside and outside corporate environments. The need to identify, manage, control and secure a quickly growing web of connections and outside devices is making the already challenging task of security even more important, and onerous. In his session at @ThingsExpo, Rich Boyer, CISO and Chief Architect for Security at NTT i3, discussed new ways of thinking and the approaches needed to address the emerging challenges of security i...
While the focus and objectives of IoT initiatives are many and diverse, they all share a few common attributes, and one of those is the network. Commonly, that network includes the Internet, over which there isn't any real control for performance and availability. Or is there? The current state of the art for Big Data analytics, as applied to network telemetry, offers new opportunities for improving and assuring operational integrity. In his session at @ThingsExpo, Jim Frey, Vice President of S...
"DX encompasses the continuing technology revolution, and is addressing society's most important issues throughout the entire $78 trillion 21st-century global economy," said Roger Strukhoff, Conference Chair. "DX World Expo has organized these issues along 10 tracks with more than 150 of the world's top speakers coming to Istanbul to help change the world."
"We provide IoT solutions. We provide the most compatible solutions for many applications. Our solutions are industry agnostic and also protocol agnostic," explained Richard Han, Head of Sales and Marketing and Engineering at Systena America, in this SYS-CON.tv interview at @ThingsExpo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We are focused on SAP running in the clouds, to make this super easy because we believe in the tremendous value of those powerful worlds - SAP and the cloud," explained Frank Stienhans, CTO of Ocean9, Inc., in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We've been engaging with a lot of customers including Panasonic, we've been involved with Cisco and now we're working with the U.S. government - the Department of Homeland Security," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held June 6-8, 2017, at the Javits Center in New York City, NY.
The financial services market is one of the most data-driven industries in the world, yet it’s bogged down by legacy CPU technologies that simply can’t keep up with the task of querying and visualizing billions of records. In his session at 20th Cloud Expo, Karthik Lalithraj, a Principal Solutions Architect at Kinetica, discussed how the advent of advanced in-database analytics on the GPU makes it possible to run sophisticated data science workloads on the same database that is housing the rich...
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
What sort of WebRTC based applications can we expect to see over the next year and beyond? One way to predict development trends is to see what sorts of applications startups are building. In his session at @ThingsExpo, Arin Sime, founder of WebRTC.ventures, discussed the current and likely future trends in WebRTC application development based on real requests for custom applications from real customers, as well as other public sources of information.
"The Striim platform is a full end-to-end streaming integration and analytics platform that is middleware that covers a lot of different use cases," explained Steve Wilkes, Founder and CTO at Striim, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
SYS-CON Events announced today that Massive Networks will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Massive Networks mission is simple. To help your business operate seamlessly with fast, reliable, and secure internet and network solutions. Improve your customer's experience with outstanding connections to your cloud.
"MobiDev is a Ukraine-based software development company. We do mobile development, and we're specialists in that. But we do full stack software development for entrepreneurs, for emerging companies, and for enterprise ventures," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.