Product Testing During Product Development

Product Testing: How HardTech Startups Can Get Started

Physical product developers test their product designs for a number of reasons: (1) to ensure and protect customers' health and safety; (2) to ensure the product fulfills the customer value proposition in a quantifiable and repeatable way; (3) to verify the design and construction actually achieves its goals; (4) to confirm that production processes and components meet the original design goals; and/or (5) to be used as a formal completion criteria for delivery of outsourced design or manufacture.

Regardless of what your product testing is focused on, you don’t want to be unprepared for this phase of product development, so let’s review how to design and execute a comprehensive product test plan. In the end, as you successfully define your product category, your Test Plan will become a valuable piece of intellectual property.

Creating a Test Plan 

A test plan is a written document that defines how you plan to test your product for performance (i.e. what does your product do? How does it perform?), quality (how reliably does your product DO the thing it's intended to do?), safety and compliance (does your product hurt anyone while the product does what it's supposed to do?).   

When creating your test plan, outline which feature(s) you need to test. List out the multiple use cases that need to be formally captured. Also list the various ways the product can be misused. It's vital to include misuse considerations in your test plan, as well as shipping, installation, and maintenance / servicing scenarios that may or may not affect your product's lifetime and warranty details.  In cases where you are partnered with a large or sophisticated customer, this may be a jointly developed deliverable. For more complicated or multi-product situations, the test plan may reference specific test procedures that are documented separately. 

Your test plan should define the methodology or approach you'll take, all equipment you'll need to execute the test(s), and any acceptance criteria for pass/fail conclusions. Compliance or regulatory test criteria, which are the requirements that a product must meet in order to be accepted by a certifying / jurisdictional authority, should also be included. Note that not all acceptance criteria may be known when you create your initial test plan - and that's OK. You don't have to know all the details right off the bat. This test plan is something you create early on and update as as you learn more. Remember, the process is iterative. And you don’t have to start from scratch – mHUB shares a Test Plan template that you can use to get started.

Testing for Performance

You have to make sure the product actually works, so define those performance parameters. Detail how the product performance responds to your specific customers' needs and use. Remember those insights you gathered from the 'voice of customer' research you did? That information comes into play for performance testing. Another source of potential can be benchmarks to compare your product's performance to your competition. Some attributes could include strength, speed, comfort, user interface / user experience (UI/UX), and other specific or alternate features. Whenever possible look for existing written procedures from ASTM, SAE, ISO, ANSI, IEEE, or other standards or industry organizations.  

When to Test Product Performance:  
• Early in development to make sure your product is feasible 
• Final prototype to design completion 
• First prototype from supplier to confirm alignment to design 
• First product off custom tooling 
• Each lot produced until product / process is stable 

In order to eliminate confusion and wasted effort, performance testing should be repeatable and consistent.  

As your design and market knowledge progress, capture your learnings in the test plan. Document design elements that were improved along the way (geometry, forces, etc.), lessons learned from field failures, and customer feedback into your evolving test plan. Another important source of input to a mature design and test plan is a a Design Failure Mode Effects Analysis (DFMEA) which helps you identify in a methodical way any potential risks as a result of a  your design  decisions. Discussion of risks naturally lead us to testing for quality. 

Testing for Quality

While “quality” is the customer’s subjective experience of how your product fulfills their expectations, we can anticipate the drivers of the perception and test for success in meeting them. Customers expect their product  to be “durable” and/or that the product works "easily” or “enjoyably". Their experience is driven by the conditions the product is to be used in. Is it going to be used inside or outside? In hot, cold, dry, or wet weather? Will it be used during the daytime or in darkness? Does the product stay in one place (stationary) or is it mobile? What about the target user? What is their age and capability range? Male, female, or is gender not a factor at all? Are they short, tall, slim or robust? How about usage? Will this product be used for minutes or hours? Continuous or intermittently? Daily, weekly, monthly?  During handling, packaging, and transportation, how will this product hold up? Will varying ranges of vibration ‘break’ your product? Everything breaks – so understanding those thresholds is critical. 

Fundamentally, you are working to control for variability in the way your product is built, the parts that go into it, the conditions it’s used in, and the ways that it is used. 

When to Test Product for Quality: 

• Early prototypes only as needed to identify design challenges 
• First product with production parts and tooling 
• At major design changes Each lot after until product/process is stable 
• Continual through production – spot check to document continued compliance 
• With a change in suppliers, in case of new material (cost reduction) and/or new process  

Note that it’s appropriate to perform a subset of targeted tests tailored to the specific changes in design / tooling / components. 

Quality costs money. Make sure that the criteria and testing are tightly aligned to your value proposition. 

Testing for Compliance

Compliance testing is not optional. Compliance testing is critical to keeping people safe! Product developers must consider safety hazards like chemical exposure and potential for electrical shock or arcing. Electrical arcing is when electricity jumps from one connection to another; this flash of electricity can reach temperatures of 35,000°F and can cause fires and serious burns / injuries. If you’re developing hardtech, you will most likely have an electrical component, so here’s a PRO TIP: Design with Recognized Components (UR), especially for electrical components. For example, if you are incorporating lithium batteries, you need to ensure that they are properly protected against thermal runaway. 

In many cases, products must be compliant with city, county, state, and federal regulatory bodies, in addition to jurisdictional bodies like hospitals, hotels, or dormitories. Your design should consider regulatory and safety early on. To better understand the compliance standards you need to meet and what compliance testing you might need to do, it's helpful to do some research. Check UL's Online Directory of Standards and trade associations like the National Fire Protection Association (NFPA). You can get info on local building codes online from the Consumer Product Safety Commission (CPSC), HVAC requirements from the Air-Conditioning, Heating, and Refrigeration Institute (AHRI), and food-safe standards from the U.S. Food & Drug Administration (FDA).  

When to Test Product for Compliance 

• Identify relevant standards and design compliance into your product early-on 
• First product off custom tooling if possible 
• Continual through production – spot check to document continued compliance 
• Confirm certification markings (UL/UR) are sufficient for application 
• Your documents to supplier MUST state UL Listing requirement 
• Work with agency (UL/CSA/ETL) to confirm ‘passing’ prior to shipping to customers 

CONCLUSION 

Embrace failures! Your prototypes will be broken during the product testing phase; no failures means no learning. After a test failure, you or members of your team should determine if requirement(s) were set correctly or if there is a design issue or manufacturing defect. Check that parts match the drawings, and work with your supplier to modify the material(s), tolerance(s), geometry, and/or change component(s).

When it comes to failure, accept that failure will happen – all the time, to every team, regardless of your best calculations. Just make sure you make a positive change so that it doesn’t happen again.

Finally, testing to failure helps you, the product developer, identify “over engineering”. Don’t put cost in your product that your customer doesn’t need or pay for. You can also work with hardtech mentors in the mHUB ecosystem to help you through the product failure that is inevitable during the product testing phase. 

Become an mHUB Member so you can connect with a hardtech mentor.