A Pilot Is Not a Demo: Designing IoT Field Validation That Produces Decisions
A demo proves that a team can make a system work once under controlled conditions. A pilot must answer a harder question: can the product, people and operating process work together over time in a real environment? The gap is especially wide in IoT. Hardware meets weather, radio interference, inconsistent installations, unreliable networks, energy constraints and customers who are not part of the development team. A useful pilot is therefore not an extended sales event. It is a decision instrument: a designed experiment that states what must be learned, which evidence will be sufficient and what the team will do if the result is negative.
Define the decision before the installation
A common mistake is to begin with the number of devices, the site and a launch date. Those are execution details, not the purpose of the pilot. Start by naming the decision the work must enable: whether the architecture fits the environment, whether operating savings justify deployment, whether the product can be maintained at scale, or whether a defined user group receives enough value. A good decision includes real alternatives—continue, change, narrow or stop. Then write three to five high-risk hypotheses and give each one a metric, data source and threshold. This prevents the familiar outcome in which everybody was impressed but nobody knows whether to allocate budget. A pilot does not need to prove the entire vision. It needs to reduce the uncertainty that blocks the next responsible decision.
Choose a site that represents real difficulty
A site that is too convenient produces success that may not generalise. An extreme site can destroy an immature product before the team learns anything useful. Selection should represent the expected variation in building materials, distance, radio load, temperature, moisture, power availability, usage patterns and local support. Sometimes several small zones are better than one uniform site because they expose different edges without dramatically increasing volume. Document what the site represents and what it does not. That note keeps local success from becoming a global promise. The NIST Municipal IoT Blueprint published in July 2019 distinguishes pilot networks used to evaluate solutions and use cases from broad production deployments. The product implication is simple: a pilot may be limited, provided that the limits of inference are visible.
Separate device health from service quality
A green LED or heartbeat proves that a device is alive; it does not prove that the service works. Measure the complete chain. Does the sensor produce a plausible reading? Does the packet arrive? Is it decoded correctly? Does the event enter the system on time? Can a user act on it? Does the action create an outcome? Technical measures such as packet delivery, latency, reboots and energy consumption must sit beside product measures such as insight availability, response time and completed tasks. When the layers remain separate, a team can tell whether a failure originated in radio, backend, installation or the human workflow. Without that distinction, one average availability percentage hides an entire chain of failure modes and encourages investment in the wrong fix.
Measure radio over time and by layer
An RSSI screenshot taken on installation day is not a communications test. RF performance changes by hour, traffic, doors opening, nearby equipment, vegetation, weather and configuration. A pilot should collect distributions rather than a single value: signal strength, link quality, retransmissions, packet loss, rejoin time and mesh paths where applicable. Firmware, frequency, transmit power, antenna, location and gateway version must be recorded so that change can be explained. Deliberate tests—disconnecting a gateway, switching a channel or blocking a route within safe limits—show whether the system recovers rather than merely surviving an ordinary day. The business ultimately cares about the service enabled by the link, but layered telemetry is what lets engineering improve that service efficiently.
Treat energy as a model, not a battery percentage
For a mains-powered device, a short outage may become an availability problem. For a battery product, a small error in the transmission cycle can double field visits. Measure current in sleep, sensing, processing, transmission, retry and firmware-update states, then combine those measurements with a realistic usage profile. A battery-life forecast should include temperature, ageing, cell variation and exceptional events rather than one typical laboratory number. The pilot should assign an energy budget to each operation and check whether poor connectivity keeps a device awake. The model must also produce a business decision: replacement interval, visit cost, spare inventory and customer risk. Battery life is a product and operating characteristic, not merely a component specification.
Provisioning reveals whether deployment can scale
A development team can connect ten units manually, inspect logs and repair a bad identifier. A field installer needs a different flow: unambiguous identity, site assignment, authorisation, link check, firmware confirmation and proof that the first data arrived. Measure installation time, first-attempt success, manual interventions and error types. If every device needs an engineer, the pilot may function but it has not demonstrated deployability. A complete flow also covers replacement, transfer between customers, secure reset and decommissioning. The goal is not to hide complexity; it is to translate complexity into clear steps with feedback and recovery. How a device enters the system largely determines whether the business can support one hundred devices or ten thousand.
Turn security principles into observable scenarios
NISTIR 8259A, published May 29, 2020, defines an IoT device cybersecurity capability baseline that includes device identification, configuration, data protection, restricted interface access, secure software update and cybersecurity-state awareness. ETSI EN 303 645 V3.1.3, published in September 2024, provides a broader baseline for consumer IoT products. A pilot should turn these principles into observable scenarios: attempting to enrol an unauthorised unit, rotating a key, revoking access, applying a signed update, reconnecting after loss of network and reporting a vulnerability. There is no need to perform a dangerous attack in a customer environment, but controls and operating procedures should be tested together. Security left outside the product lifecycle becomes costly debt precisely when the device population begins to grow.
Measure recovery, not only uptime
A field system will experience power loss, an unavailable gateway, a slow service, clock drift, an expired certificate or an interrupted update. A pilot that hides faults discards its most valuable learning opportunity. Inject failures within safe boundaries and measure detection, isolation, recovery and data completion. Does the device buffer readings locally? Does it prevent duplicates? Can the platform distinguish a missing observation from a value of zero? Does the operator receive an alert that explains what to do? High availability can coexist with poor recoverability when one rare failure requires a physical visit. Mean time to recovery, automatic recovery rate and observations lost may tell a more useful story than a rounded uptime percentage.
Expose the operating cost
A functioning unit is not a business model. Record the time spent on site planning, installation, support, anomaly review, hardware replacement and firmware maintenance. Separate one-time learning from work repeated for every customer and device. Founder-led support can be appropriate while learning, but the measurement should show what happens when another team performs it. Total cost includes gateways, connectivity, cloud, storage, visits, inventory, warranty and replacement—not only the bill of materials. Value must be equally concrete: hours saved, earlier decisions, events avoided or service quality improved. A useful pilot puts reliability, value and cost on the same page so that technical enthusiasm cannot conceal operating economics.
Build the pilot as an evidence pipeline
Data that cannot be reconstructed will eventually become a presentation of opinions. Every metric needs a definition, unit, time window, source, software version and exclusions. Store configuration and installation events beside telemetry because a location or firmware change can explain a sudden shift. Decide who may change a threshold, how missing observations are marked and how manual intervention is recorded. A dashboard supports monitoring, but a decision document also needs comparison with a baseline, segmentation by site and treatment of outliers. The aim is not to collect everything. It is to preserve the evidence needed to test each hypothesis. A pilot is a temporary knowledge product; it should leave behind decisions that remain explainable to somebody who was never on site.
Use gates instead of a closing ceremony
Ending on schedule is not success. Define decision gates before starting: Go when value and reliability meet the threshold and remaining gaps are tractable; Iterate when the core is promising but a specific change is required; Pause when evidence is missing; Stop when the central assumption fails or the economics do not work. Stopping is not failure when it prevents a bad deployment. Give every open gap an owner, budget and decision date rather than an unlimited wish list. If the project continues, the pilot should produce a production-transition plan covering supported configuration, capacity, monitoring, service levels, security, support, training and end-of-life policy. The transition is not simply more units. It is a change in the system of responsibility.
The real output is a better decision
A strong pilot can end in yes, no or yes under conditions. Its quality is not measured by a photograph of installed devices, but by how much uncertainty disappeared and how clearly the remaining uncertainty is expressed. Work on IoT products, including documented projects in Dor Arad’s portfolio such as DNG Technologies, demonstrates why hardware, network, software, operations and business model must share one decision language. No single layer proves readiness. When a decision, scenario, metric and threshold are connected from the beginning, the pilot stops being a technology performance and becomes a management tool. It reveals not only whether the device works, but whether a complete system can deliver value, remain secure and improve after the development team leaves the site.