All About Functional Testing of IoT Cloud Applications

Key takeaways:

  • Business case for functional testing
  • What does functional testing of IoT Cloud applications include
  • Shortcomings of current approaches
  • Solution
Doppelio IoT Functional Testing

Functional testing of IoT Cloud Applications can be a daunting task given the complexity and variety of “things”, applications, and scenarios that need to be tested. This article focuses on the need for functional testing, what it includes, and the shortcomings in the current approaches followed by the testing teams.

 

Business Case – why it is critical to do that?

Functional testing is important to ensure that the application is behaving as expected for different scenarios arising out of the things and environments being monitored. Specific scenarios of the business case include:

  • Ensuring right computations – OEE, Utilization, and predicted failures
  • Alerts and events not generated on time can have high-impact consequences
  • Lack of right visualization and reporting of the data can lead to incorrect decisions
  • Impact to platform adoption due to user’s distrust on the app functionality
  • Device management and remote fixes with functionalities like FOTA, in which cases a failure could lead to field visits/recall for servicing
  • Ensuring that the edge and cloud integration works fine beyond just the protocol and payload agreements to avoid last-minute surprises in the field tests or sometimes only on the field deployments

 

What does functional testing include in the context of IoT Cloud Applications?

Scenario testing:

  • User actions on the applications
  • Data and event coming from the devices
  • User actions spanning across the applications on the cloud and the device

Application areas to be validated:

  • Web app
  • Mobile app
  • API
  • Database testing
  • Device Management
  • Data Ingestion Scenarios
  • Real time data processing
  • Batch data processing
  • Reconciliation of old stored data coming from devices

 

What are possible shortcomings that one might have in the current approaches?

Testers need the capabilities below to do a good job of testing

  • Data flow for happy and unhappy paths/scenarios
    • Includes sequencing of different kind of payloads
    • Data values for one or more related parameters impacting the functionality
    • Data values over one or many iterations
    • Data values going beyond or below thresholds
    • Data values getting misses or becoming junk
  • Communication characteristics arising out of network conditions
    • Payloads arriving late
    • Payloads getting missed out
    • Duplicate payloads being received
    • Payloads arriving in different order
  • System level behavior of the intelligent edge and cloud application for the different aspects of distributed systems like split brain and so on

Testers use the following options today

  • Test with physical devices in the lab
    • Different data scenarios that be generated with a physical device is very limited, especially for unhappy paths
    • Sometimes specialized knowledge and physical signal generators are needed to do the testing
    • It takes a lot of time
    • Repeatability and consistency are an issue because of the manual activity
  • Use protocol simulator tools to manually generate data and payloads to send the data
    • Very laborious activity, where the tester is forcing himself to behave like a device
    • Take a lot of time, patience, and attention to do a good job of testing
    • The very nature of the activity demotivates, limits the testers to cover the range of scenarios they would ideally like to cover
    • Repeatability and consistency are an issue because of the manual activity
  • Build custom simulators and scripts
    • This option could be a real solution that solves the requirements ideally
    • But the complexity of building these itself can be a challenge and needs good programming skills and knowledge of the nitty gritty of the IoT Protocol and the Platform.
      • Even then, things like simulation of the device behavior for network conditions, bidirectional data communications, etc., can be very difficult to conquer.
      • This means the testers are spending valuable time on an enabler instead of focusing on the core activity of validating features and building automation.
      • In the times when the software engineering resource supply is scare and costly, this isn’t an ideal situation for the business.
    • Even if the 1st version of simulator is created, maintaining it continuously over the lifetime of the product (AUT) across releases can become very cumbersome.
      • Some studies show that the testers spend more than 60% of time just maintaining the tests even in traditional application cases, this can only go higher for the IoT scenarios.
    • Miss out on the end-to-end validations required in cases of architectures that includes intelligent edge applications. Most teams don’t even realize this requirement to start with – thinking that the comprehensive validation of cloud and device applications separately for the functionalities is enough.
      • The fallacies of distributed systems are very much applicable to the IoT architectures and that needs to be tested for.

Doppelio – Purpose-built to test IoT applications

Doppelio’s patent-pending approach leverages the concepts of virtualization and simulation to give you a testbed that is a very close replica of the real world. Test scenarios are easy to create with Doppelio Web Application without having to write even a single line of code.

Click here for more details on how Doppelio approaches functional testing of IoT cloud applications. Write to us if you want to see a quick demo.

Rajesh K Doppelio

Rajesh K, Co-Founder & Head of Products