Welcome!

@ThingsExpo Authors: Pat Romanski, Carmen Gonzalez, Yeshim Deniz, Elizabeth White, Ruxit Blog

Related Topics: @ThingsExpo, Java IoT, Agile Computing, Wearables, @CloudExpo

@ThingsExpo: Blog Feed Post

You Can’t Test All the 'Things' | @ThingsExpo #IoT #M2M #API #InternetOfThings

Young engineers building all sorts of IoT devices without a thought to protocol or integration standards

You Can’t Test All the Things: API, IoT, ROI TBD

I have three words for everyone in software testing: prioritize, prioritize, and prioritize.

You can't test every possible permutation of your software, especially so with APIs and IoT devices where you're placing much of the user experience in the hands of integrators to your core products and services. You just can't, so we should throw our hands up now and just give up, right?

What sort of tester are you anyway?

I'm not; at best I'm an automation engineer. I don't nearly have the mind or patience that is required of a full-time, dedicated testing professional. But I talk to them and lots of other people every day, and I'm still a developer by heart, so I feel very kin to everyone involved in the software delivery process.

You don't have to be a tester to understand the constraints of being human. Humans make mistakes and only have so much time per day. Developers who are starting to experience a shift-left in testing responsibilities might make sure that all functions and procedures are unit tested, but often don't go much further simply because there isn't enough time. Not enough people, not enough time...two things I hear an awful lot and part of the reason why I'm in the automated testing tools business with Ready! API.

Side note: I would also like to take this opportunity to say that "no budget" and "not a priority" are the worst reasons not to invest in a better future, one where automation gives you back time to build better software and where getting your delivery pipeline right is critical to moving forward with rapid iteration.

But shouldn't we always test more?

No. Well, yes. Actually, sort of and not always. There's an aspect of diminishing return to introducing more testing in that if you don't prioritize the most business-critical components and workflows to be tested first, you fail your job, your business, and yourself as a professional tester.

We simply can test everything, we just don't have the time, and we also need to stay nimble. If I were a tester, I'd probably get fired for wanting to confirm that what I was told to work on actually affects the bottom line of the business in the most impactful way at this moment in time. I would not sit there creating tests just so that we could report that we have 79% instead of 78% testing coverage today. I would want to see ROI, even though apparently "there ain't no ROI in software testing".

What about all the new stuff, shouldn't that be tested?

I got a question after my talk at APIdays IoT San Francisco 2015 in July: "If you don't know how people are going to combine your devices and APIs, how can you ensure that you've tested everything?" In no disrespectful way, this kind of question underlines the immaturity of the IoT space. We've got a very young startup culture (think 20-somethings with multiple PhDs) attacking big problems, most usually around how the digital world imbues itself back into the physical world, as quickly as possible for as cheap as possible.

There's going to be some broken things, incomplete thoughts, and definitely failures to learn from. I saw this first hand at O'Reilly Solid the week after APIdays too. The speakers were experienced industry leaders, and it was good that those people were on stage speaking to the young engineers building all sorts of IoT devices without a thought to protocol or integration standards. The median booth sponsor was grad school geniuses that are practically DevOps natives building our future...so busy implementing that they may not even know to ask the more important question, "should we"?

What do you have against startups?

Nothing personal, I've been involved in a few myself. The "startup kids" are working incredibly hard, many harder than I've ever worked, to make it or break it in the cutthroat technology market that we've built for ourselves. And when I say "kids", I mean figuratively; folks who bring a variety of backgrounds, skills, and experience to a startup still often have a lot to learn about running a lean operation before things lock in and start whirring.

Some things take time to learn (thanks to our lizard brains) and we (the technology industry) would do well to remember that people are still naturally slow to accept and properly utilize new technologies (like for instance the slow but increasing industry adoption of lightweight service virtualization tools).

If there was one thing I'd take issue with related to startup culture, it's that some important corners can get cut when you're so busy accelerating that you don't have time to ask "should we?". For instance, API security is a huge deal now because of a number of very public API security failures have occurred in the past few years. Moonpig, Uber, Tinder, and others could have avoided this just by applying free knowledge to their designs.

The stakes get bigger too the closer to money and safety your software is, like as in financial and medical institutions, or worse defense and air travel; proactively identifying vulnerabilities to critical services is a fundamental reason for regulatory audits, and keep omissions in safety introduced by startup culture thinking from having a massive negative affect on everyday life for the average digital citizen.

Then how do I know when I've tested enough?

That's up to you, it's your software, it's your customers and their expectations that should help you know when enough is enough. In the absence of formal UX and operational metrics, you may need to set some goals starting with your technical team. You should also be talking directly with your consumers, getting their feedback and input on what they expect from your software.

Go read some James Bach or Michael Bolton books if you want to learn about testing in general, but when it comes down to specifics like SLAs and reporting to management, there are very tangible questions to ask about quality. For instance, a slightly more matured set of questions around testing IoT and other multiples of design complexity (like Microservices, Hypermedia, etc.) might include:

  • Should my goal be to test all the things and permutations?
  • What do the user-experience metrics say about what my customers see as a priority?
  • What technical omissions contribute to our worst business deficits?
  • Do I need to build/test/ship this, right now, at this very moment?
  • What do we gain and/or lose by shipping what we have, such as it is?

Okay, so I don't have to test more?

Don't we all wish; sorry, no, this isn't a free lunch. Yes, you should be testing more, almost universally. Most software vendors and engineers know that automated testing is the only way to achieve high quality with less manual intervention, but haven't seen the pain that a lack of proper testing causes.

Good engineers and technicians have consciences and know when something needs more attention, which is why I leave it up to you to answer the question "how much testing do I need?" You don't need your inner voice telling you that you need to test and monitor something that only you know would cripple your business if it went down for any amount of time. So in those moments, I humbly submit to you that if you see or feel the future pain, it is your responsibility to write the test, or at least put it in the priority backlog, or something that your team is sure to follow up on.

Often backlogged is "test coverage", a useful assessment tool, but also just a broad-spectrum approach to getting visibility over technical gaps in software delivery. It helps you see when new things are implemented without the proper team communication, but should not be a goal on its own right. When was the last time your customer said "if you don't have at least 82% testing coverage, I don't want to use your software"? Right. Most consumers care about the downstream effects of bad software like app crashes, not the things you do to prevent bugs from getting to production. Coverage is not a golden idol, it's just a method of assessing where your weaknesses lie.

Like security testing, test coverage is a measurement of only one aspect of readiness that should be part of a larger perspective on customer-focused software quality. The connection between testing and customer value shows up in other areas like beta acceptance testing, performance monitoring, and consumer feedback. It's there for the taking when we want it.

That's my experience with testing "all the things", but what's yours? I'd love to hear your thoughts.

Read the original blog entry...

More Stories By SmartBear Blog

As the leader in software quality tools for the connected world, SmartBear supports more than two million software professionals and over 25,000 organizations in 90 countries that use its products to build and deliver the world’s greatest applications. With today’s applications deploying on mobile, Web, desktop, Internet of Things (IoT) or even embedded computing platforms, the connected nature of these applications through public and private APIs presents a unique set of challenges for developers, testers and operations teams. SmartBear's software quality tools assist with code review, functional and load testing, API readiness as well as performance monitoring of these modern applications.

@ThingsExpo Stories
20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
Bert Loomis was a visionary. This general session will highlight how Bert Loomis and people like him inspire us to build great things with small inventions. In their general session at 19th Cloud Expo, Harold Hannon, Architect at IBM Bluemix, and Michael O'Neill, Strategic Business Development at Nvidia, discussed the accelerating pace of AI development and how IBM Cloud and NVIDIA are partnering to bring AI capabilities to "every day," on-demand. They also reviewed two "free infrastructure" pr...
Have you ever noticed how some IT people seem to lead successful, rewarding, and satisfying lives and careers, while others struggle? IT author and speaker Don Crawley uncovered the five principles that successful IT people use to build satisfying lives and careers and he shares them in this fast-paced, thought-provoking webinar. You'll learn the importance of striking a balance with technical skills and people skills, challenge your pre-existing ideas about IT customer service, and gain new in...
SYS-CON Events announced today that Hitrons Solutions will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Hitrons Solutions Inc. is distributor in the North American market for unique products and services of small and medium-size businesses, including cloud services and solutions, SEO marketing platforms, and mobile applications.
The Internet of Things will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, demonstrated how to move beyond today's coding paradigm and shared the must-have mindsets for removing complexity from the develop...
For basic one-to-one voice or video calling solutions, WebRTC has proven to be a very powerful technology. Although WebRTC’s core functionality is to provide secure, real-time p2p media streaming, leveraging native platform features and server-side components brings up new communication capabilities for web and native mobile applications, allowing for advanced multi-user use cases such as video broadcasting, conferencing, and media recording.
In his session at @ThingsExpo, Steve Wilkes, CTO and founder of Striim, will delve into four enterprise-scale, business-critical case studies where streaming analytics serves as the key to enabling real-time data integration and right-time insights in hybrid cloud, IoT, and fog computing environments. As part of this discussion, he will also present a demo based on its partnership with Fujitsu, highlighting their technologies in a healthcare IoT use-case. The demo showcases the tracking of patie...
Almost two-thirds of companies either have or soon will have IoT as the backbone of their business. Though, IoT is far more complex than most firms expected with a majority of IoT projects having failed. How can you not get trapped in the pitfalls? In his session at @ThingsExpo, Tony Shan, Chief IoTologist at Wipro, will introduce a holistic method of IoTification, which is the process of IoTifying the existing technology portfolios and business models to adopt and leverage IoT. He will delve in...
SYS-CON Events announced today that Outlyer, a monitoring service for DevOps and operations teams, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Outlyer is a monitoring service for DevOps and Operations teams running Cloud, SaaS, Microservices and IoT deployments. Designed for today's dynamic environments that need beyond cloud-scale monitoring, we make monitoring effortless so you...
Unsecured IoT devices were used to launch crippling DDOS attacks in October 2016, targeting services such as Twitter, Spotify, and GitHub. Subsequent testimony to Congress about potential attacks on office buildings, schools, and hospitals raised the possibility for the IoT to harm and even kill people. What should be done? Does the government need to intervene? This panel at @ThingExpo New York brings together leading IoT and security experts to discuss this very serious topic.
It is one thing to build single industrial IoT applications, but what will it take to build the Smart Cities and truly society changing applications of the future? The technology won’t be the problem, it will be the number of parties that need to work together and be aligned in their motivation to succeed. In his Day 2 Keynote at @ThingsExpo, Henrik Kenani Dahlgren, Portfolio Marketing Manager at Ericsson, discussed how to plan to cooperate, partner, and form lasting all-star teams to change the...
The buzz continues for cloud, data analytics and the Internet of Things (IoT) and their collective impact across all industries. But a new conversation is emerging - how do companies use industry disruption and technology enablers to lead in markets undergoing change, uncertainty and ambiguity? Organizations of all sizes need to evolve and transform, often under massive pressure, as industry lines blur and merge and traditional business models are assaulted and turned upside down. In this new da...
“We're a global managed hosting provider. Our core customer set is a U.S.-based customer that is looking to go global,” explained Adam Rogers, Managing Director at ANEXIA, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
WebRTC services have already permeated corporate communications in the form of videoconferencing solutions. However, WebRTC has the potential of going beyond and catalyzing a new class of services providing more than calls with capabilities such as mass-scale real-time media broadcasting, enriched and augmented video, person-to-machine and machine-to-machine communications. In his session at @ThingsExpo, Luis Lopez, CEO of Kurento, introduced the technologies required for implementing these idea...
In an era of historic innovation fueled by unprecedented access to data and technology, the low cost and risk of entering new markets has leveled the playing field for business. Today, any ambitious innovator can easily introduce a new application or product that can reinvent business models and transform the client experience. In their Day 2 Keynote at 19th Cloud Expo, Mercer Rowe, IBM Vice President of Strategic Alliances, and Raejeanne Skillern, Intel Vice President of Data Center Group and G...
Financial Technology has become a topic of intense interest throughout the cloud developer and enterprise IT communities. Accordingly, attendees at the upcoming 20th Cloud Expo at the Javits Center in New York, June 6-8, 2017, will find fresh new content in a new track called FinTech.
@GonzalezCarmen has been ranked the Number One Influencer and @ThingsExpo has been named the Number One Brand in the “M2M 2016: Top 100 Influencers and Brands” by Onalytica. Onalytica analyzed tweets over the last 6 months mentioning the keywords M2M OR “Machine to Machine.” They then identified the top 100 most influential brands and individuals leading the discussion on Twitter.
20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
Manufacturers are embracing the Industrial Internet the same way consumers are leveraging Fitbits – to improve overall health and wellness. Both can provide consistent measurement, visibility, and suggest performance improvements customized to help reach goals. Fitbit users can view real-time data and make adjustments to increase their activity. In his session at @ThingsExpo, Mark Bernardo Professional Services Leader, Americas, at GE Digital, discussed how leveraging the Industrial Internet and...
SYS-CON Events announced today that delaPlex will exhibit at SYS-CON's @CloudExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. delaPlex pioneered Software Development as a Service (SDaaS), which provides scalable resources to build, test, and deploy software. It’s a fast and more reliable way to develop a new product or expand your in-house team.