NeedForHeat DataGear: Pre-Transition Heating Insights

• NeedForHeat DataGear is an open data collection system designed for pre-transition residential heating analytics by capturing high-quality time series data.
• It integrates modular ESP32-based devices with cloud feeds to monitor parameters like boiler temperatures, CO₂ levels, and smart meter readings with scientific accuracy.
• The system supports empirical diagnostics and retrofit planning through reliable measurement, privacy-preserving data handling, and tailored campaign configurations.

Searching arXiv for the cited NeedForHeat DataGear and related district-heating/data-quality papers. arXiv search query: (Hofte et al., 8 Sep 2025) NeedForHeat DataGear NeedForHeat DataGear is an open hardware and open software data collection system designed to accelerate the residential heating transition by acquiring time series monitoring data in homes that have not yet undergone a heating transition. It targets pre-transition residential dwellings that still use gas-fired boilers and have not yet switched to heat pumps, with the explicit aim of capturing the empirical thermal characteristics of the building and heat distribution system, current heating system behavior and efficiency, and residents’ comfort needs before retrofit or system change. The system is presented as modular, adaptable, cost-effective, and user-friendly, and is positioned against home automation platforms, post-installation monitoring, and electricity-centric tools by prioritizing scientific-grade, reliable time series collection and analysis with accurate timestamps, explicit units, and consistent schemas (Hofte et al., 8 Sep 2025).

1. Scope and conceptual position

NeedForHeat DataGear is intended for researchers and energy professionals or coaches running monitoring campaigns, screening homes, and preparing tailored advice. Its focus is explicitly pre-transition: the system is designed to evaluate the current situation in existing homes before any retrofit or switch to a heat pump. In that role, it collects heating-relevant parameters that include boiler flow and return temperatures, CO₂ for occupancy and ventilation inference, gas and electricity via smart meter P1, thermostat and boiler states via OpenTherm, and external weather and cloud feeds (Hofte et al., 8 Sep 2025).

The system is differentiated from several adjacent categories. Unlike home automation platforms such as Home Assistant and Homey, it is built for scientific-grade, reliable time series collection and analysis rather than real-time control or convenience automation. Unlike post-installation monitoring systems such as EPV or NOM-keur compliance systems, it emphasizes pre-transition diagnostics to inform feasibility, sizing, and prioritization of upgrades. Unlike smart thermostats and electricity-centric tools, it aims to capture a wider set of heating-relevant parameters (Hofte et al., 8 Sep 2025).

A common misconception is to treat DataGear as a building automation or actuation platform. The documented design instead centers on monitoring-only acquisition: OpenTherm interception is explicitly described as “no control injection; monitoring-only,” and the server is organized around ingestion, storage, and analysis workflows rather than household control logic (Hofte et al., 8 Sep 2025). This suggests that its primary scientific role is observational and diagnostic rather than supervisory.

2. Architecture and data flow

The end-to-end architecture consists of in-home measurement devices, cloud feeds, a smartphone app for resident interaction, optional batch imports, and a cloud-agnostic server stack. Measurement devices in the home send time series via the home’s Wi‑Fi to the NeedForHeat Server. Three primary device classes are identified: the Living Room Module, the Smart Meter Module, and Boiler Modules. Cloud feeds provide additional data such as Enelogic ODA for smart meter readings, KNMI weather via API, and Remeha boiler or thermostat data. The GearUp smartphone app guides residents through provisioning, authorizes cloud feed access, and answers one-time “Energy Queries.” Batch imports, such as KNMI archives and advisor diagnostics, can be added by deployers for analysis (Hofte et al., 8 Sep 2025).

The firmware and device layer are built around ESP32-based designs, including custom PCBs and the M5Stack CoreInk platform, with communication and edge processing implemented in ESP-IDF firmware. Timekeeping uses NTP, while CoreInk additionally has a BM8563 RTC. Local and uplink communications use BLE for secure provisioning and smartphone pairing, Wi‑Fi for device-to-server communication over TLS/SSL, ESP‑NOW for local telemetry from a Boiler Monitor Satellite to a Smart Meter Module, P1 for Dutch smart meter serial data, and OAuth2-mediated cloud feeds (Hofte et al., 8 Sep 2025).

Module	Main measurements	Default sampling
Living Room Module	indoor temperature, humidity, CO₂, occupancy proxy via BLE smartphone count	every 10 minutes
Smart Meter Module	electricity kWh import/export, high/low tariffs, gas m³ via P1	every 10 minutes; gas cadence depends on DSMR version
OpenTherm Monitor	flame status, CH/DHW mode flags, modulation levels, boiler supply/return temperatures, room/setpoint temperatures	10 s, 30 s, and 5 min depending on signal
Boiler Monitor Satellite	two DS18B20 pipe-clamp sensors for flow/return temperatures	10 seconds

The upload format is JSON. Uploads contain device metadata, an upload timestamp, and an array of measurements; each measurement includes {property_name, value, time} and uses the physiquant_unit naming convention for properties. The server-side persistence layer uses MariaDB with an ER model comprising Accounts, Campaigns, Devices, DataSources, Uploads, Measurements, and Properties (Hofte et al., 8 Sep 2025).

3. Instrumentation, hardware design, and deployment configurations

The Living Room Module measures temperature, humidity, and CO₂ using the SCD41, and derives an occupancy proxy through BLE smartphone count. It is intended for placement in the main living area near the thermostat and is typically mains-powered because battery life is insufficient at the required sampling for CO₂ and BLE. The Smart Meter Module reads electricity and gas from the Dutch smart meter P1 port, records meter code and DSMR version, and supports an external P1-splitter to retain existing readers; a mains adapter is provided for DSMR3.0 meters. Boiler instrumentation is split between OpenTherm monitoring and pipe-clamp sensing. The OpenTherm Monitor captures flame status, CH/DHW mode flags, modulation levels, boiler supply and return temperatures, and room/setpoint temperatures from thermostat–boiler communication. The Boiler Monitor Satellite adds two DS18B20 pipe-clamp sensors for flow and return temperatures, and an integrated BoilerBASE module is under development (Hofte et al., 8 Sep 2025).

Calibration and timekeeping are handled pragmatically. Factory calibration of the SCD41 is used, while pipe-clamp sensors depend on good thermal contact, with pads or paste recommended. NTP synchronization occurs at provisioning and daily, and the CoreInk BM8563 RTC maintains time in low-power states. Heartbeat counters are used to verify device liveness, and upload timestamps are recorded server-side alongside device-reported measurement time (Hofte et al., 8 Sep 2025).

The documented bill of materials emphasizes low-cost field deployment. Indicative costs include €88 for a Living Room Module with CO₂, €51 for a Living Room Module without CO₂, €64 for a Smart Meter Module in one listed configuration, €19 for an OpenTherm Monitor, and €89 for a Boiler Monitor in the CoreInk BASE design, with costs noted as approximate, excluding VAT, supplier-dependent, quantity-dependent, and spanning 2021–2025. A shipping constraint is also given: an entire shipment fits a PostNL letterbox parcel with maximum dimensions of $38 \times 26.5 \times 3.2$ cm and 2 kg (Hofte et al., 8 Sep 2025).

Two deployment configurations are described. A low-cost baseline uses a Smart Meter Module, a Living Room Module without CO₂, and an OpenTherm Monitor when OpenTherm is available, with a Boiler Monitor Satellite added otherwise. A higher-accuracy build adds CO₂ to the Living Room Module, combines OpenTherm and external pipe clamps to disambiguate DHW effects, and includes P1‑BASE for robust meter interfacing (Hofte et al., 8 Sep 2025). This suggests that the platform is intended to be configurable by campaign constraints rather than fixed to a single sensor stack.

4. Software stack, data model, and data-quality handling

The device firmware is the ESP-IDF-based “NeedForHeat Generic Device Firmware.” Provisioning is performed over BLE using Espressif Unified Provisioning. A QR code encodes a device name and a Proof-of-Possession derived from the ESP32 hardware RNG, and app-assisted provisioning calls /device/activate, with the first heartbeat confirming success. Firmware behavior includes scheduling, heartbeats, power-aware timers, time synchronization, secure JSON uploads over TLS/SSL, occupancy counting through periodic Bluetooth name requests to registered MACs, and persistent buffering for eventual delivery of at least 24 hours to cover brief outages. Persistent buffering and OTA updates are implemented but not yet field-tested in large deployments (Hofte et al., 8 Sep 2025).

The backend runs as a Docker stack. The documented components are Traefik for reverse proxy and HTTPS, Portainer for orchestration or management, the NeedForHeat Server API, MariaDB, CloudBeaver, Duplicati for incremental backups to offsite storage, JupyterLab for analysis using NeedForHeat Diagnosis, and an nginx manual server for campaign manuals and FAQs. Swagger GUI is used for interactive administrative calls to REST endpoints supporting authorization, campaign and data source configuration, provisioning, uploads, and retrieval of device status (Hofte et al., 8 Sep 2025).

Data-quality handling is partly architectural and partly analytical. Units and timestamps are enforced by schema, and there is an explicit distinction between missing and zero values. Device-side buffering supports eventual delivery, while server-side checks identify overdue streams; GearUp shows last-seen timestamps to residents. The paper also notes specific filtering needs for boiler flow and return temperatures when DHW production distorts CH signals, and notes that downsampling and aggregation are performed in analysis notebooks as needed. UTC and local-time reconstruction are supported via stored time zone and H3 zone logic (Hofte et al., 8 Sep 2025).

A related visual diagnostics framework for district-heating data uses an interactive dashboard with time series plots, heatmaps, box plots, histograms, correlation matrices, and anomaly-sensitive KPIs such as skewness and anomaly detection based on modified z-scores (Christensen et al., 1 Oct 2025). This suggests that DataGear’s explicit units, timestamps, and consistent schemas are compatible with human-in-the-loop visual quality-assurance workflows of the kind used for AI-driven heat-consumption prediction.

5. Privacy, security, and governance

Privacy-by-design is a central design principle. The system states that only heating-relevant time series are collected, while “Energy Queries” are handled with local-first processing. The server stores pseudonyms and campaign IDs, and no direct personal identifiers such as addresses, emails, or names are kept in the server database. For weather handling, the resident confirms a coarse location; an H3 level 4 cell ID is sent with Gaussian noise, and the server uses the cell center for KNMI interpolation and for local/UTC time handling. For the Building Performance Signature, the app fetches BAG and 3D‑BAG, processes them locally, and sends only a derived signature rather than raw address data (Hofte et al., 8 Sep 2025).

Security controls are implemented across provisioning, transport, and cloud integration. BLE provisioning uses Proof-of-Possession to mitigate man-in-the-middle attacks, device activation uses unique tokens, all device-to-server uploads use TLS/SSL, and cloud feeds such as Enelogic use OAuth2 without exposing resident credentials to NeedForHeat. Role-restricted access is applied to pseudonym mapping, which is kept outside the NeedForHeat Server and managed by helpdesk or researcher roles (Hofte et al., 8 Sep 2025).

Governance is framed in terms of GDPR alignment, consent workflows in GearUp, and clear data scopes per campaign. Duplicati backups to secure storage such as SURFdrive or Research Drive are part of the documented infrastructure. Known risks are also explicit: cloud feed outages, diagnostic mode resets in vendor platforms, and the fact that OTA and persistent buffering still need larger-scale field validation (Hofte et al., 8 Sep 2025). This suggests that DataGear treats security and privacy as operational properties of a research infrastructure rather than as an afterthought.

6. Analytical methods, metrics, and empirical deployments

DataGear is described as the measurement backbone for physics-informed building and system analysis in NeedForHeat Diagnosis. The documented analytical core includes a single-zone RC heat-balance model for indoor air temperature,

$$C\,\frac{dT_{\mathrm{in}}}{dt} = Q_{\mathrm{in}} - UA\,(T_{\mathrm{in}}-T_{\mathrm{out}}) - Q_{\mathrm{vent}},$$

where $C$ is effective thermal capacitance, $UA$ is the heat loss coefficient, and $Q_{\mathrm{vent}}$ denotes ventilation or infiltration losses. For heating-off periods with constant outdoor temperature, the indoor decay model is

$$T_{\mathrm{in}}(t) = T_{\mathrm{out}} + \bigl(T_0-T_{\mathrm{out}}\bigr)e^{-t/(RC)},$$

with $R = 1/UA$. HLC identification is based on regression of heat input versus temperature difference over steady periods,

$$\overline{Q}_{\mathrm{in}} \approx UA\,\overline{\Delta T} + \epsilon, \qquad \Delta T = T_{\mathrm{in}}-T_{\mathrm{out}}.$$

Boiler efficiency is expressed as

$$\eta=\frac{Q_{\mathrm{out}}}{Q_{\mathrm{in}}}, \qquad Q_{\mathrm{in}} = V_{\mathrm{gas}}\cdot \mathrm{LHV},$$

and, where hydronic mass flow rate is known or estimated,

$$Q_{\mathrm{out}} \approx \dot m\,c_p\,(T_{\mathrm{flow}}-T_{\mathrm{return}}).$$

For heat-pump feasibility studies, COP is written as

$$C\,\frac{dT_{\mathrm{in}}}{dt} = Q_{\mathrm{in}} - UA\,(T_{\mathrm{in}}-T_{\mathrm{out}}) - Q_{\mathrm{vent}},$$0

Comfort assessment can use PMV/PPD when the required inputs are available, while CO₂ rise and decay analysis, together with wind and outdoor temperature from KNMI, is used to infer ventilation and infiltration contributions and air-change proxies (Hofte et al., 8 Sep 2025).

The practical HLC workflow is also specified. It begins with collecting indoor temperature from the Living Room Module, outdoor temperature from KNMI, and heat input from gas $$C\,\frac{dT_{\mathrm{in}}}{dt} = Q_{\mathrm{in}} - UA\,(T_{\mathrm{in}}-T_{\mathrm{out}}) - Q_{\mathrm{vent}},$$1 or electricity measurements. Time windows with steady-state heating and minimal DHW interference are segmented using OpenTherm flags. Interval averages of $$C\,\frac{dT_{\mathrm{in}}}{dt} = Q_{\mathrm{in}} - UA\,(T_{\mathrm{in}}-T_{\mathrm{out}}) - Q_{\mathrm{vent}},$$2 and $$C\,\frac{dT_{\mathrm{in}}}{dt} = Q_{\mathrm{in}} - UA\,(T_{\mathrm{in}}-T_{\mathrm{out}}) - Q_{\mathrm{vent}},$$3 are regressed to estimate $$C\,\frac{dT_{\mathrm{in}}}{dt} = Q_{\mathrm{in}} - UA\,(T_{\mathrm{in}}-T_{\mathrm{out}}) - Q_{\mathrm{vent}},$$4, and off-period decay segments are then used to estimate $$C\,\frac{dT_{\mathrm{in}}}{dt} = Q_{\mathrm{in}} - UA\,(T_{\mathrm{in}}-T_{\mathrm{out}}) - Q_{\mathrm{vent}},$$5, with uncertainty propagated from meter resolution, LHV variability, and sensor noise (Hofte et al., 8 Sep 2025).

Empirical deployment evidence comes from three public datasets. Twomes (2021–2022) contains 35.0 million data points across 40 homes with 35 distinct thermostat–boiler combinations. Brains4Buildings (2022–2023) contains about 386k points in office spaces. REDUCEDHEATCARB (2023–2025) contains about 30.9 million points from Remeha boilers and smart thermostats. Reported lessons include variable OpenTherm return-temperature support, the value of external pipe clamps for mitigating DHW interference, feasibility of self-installation, battery-only CoreInk lifetime of about five days in office sampling, and vendor diagnostic-mode resets that caused gaps in REDUCEDHEATCARB (Hofte et al., 8 Sep 2025). These findings place the platform within an empirical research program rather than a purely conceptual design.

7. Openness, research ecosystem, and future directions

NeedForHeat DataGear is explicitly open in both software and hardware. Software is licensed under Apache 2.0, hardware under CERN‑OHL‑P v2, and repositories are published for the generic firmware, server API, hardware designs, and diagnosis software. Public datasets linked to the platform include Twomes, Brains4Buildings, and REDUCEDHEATCARB. Contributions via pull requests are welcomed, and campaign manuals and FAQs can be customized per deployment (Hofte et al., 8 Sep 2025).

The paper also enumerates present limitations. There is no standardized Energy Performance Monitoring API or semantic schema such as Brick or Haystack. Deployment still involves manual account creation, pseudonym assignment, and installation tracking. Real-time campaign health monitoring is limited, and a NeedForHeat Cockpit is under design to automate these tasks. Persistent buffering, OTA updates, and new BASE modules require broader field validation. DHW interference in boiler telemetry remains a significant segmentation problem, and battery-only operation is insufficient for CO₂ and BLE occupancy sensing at the desired sampling rates, making mains power the recommended operating mode (Hofte et al., 8 Sep 2025).

Within the wider heat-data research landscape, DataGear occupies the field-instrumentation layer. Complementary resources operate at other scales. ThermBuild provides real-world measurements from two single-family homes and simulations of 958 TRNSYS building models, covering 960 residential multi-zone buildings in Central Europe at 15-minute resolution (Raisch et al., 1 Jun 2026). The German regionalised heat-demand dataset provides 15-minute residential heat loads and electrically covered heat load for all administrative districts in Germany (Heitkoetter et al., 2019). Country-specific space-heating threshold temperatures can be estimated from national electricity and gas consumption data through linear HDD models and AIC-based summer classification (Kozarcanin et al., 2019). This suggests a layered research workflow in which DataGear supplies pre-transition household measurements, while other datasets and methods support multi-building benchmarking, regional planning, and national-scale inference.

In that configuration, NeedForHeat DataGear can be understood as an open monitoring infrastructure for pre-transition residential heat analytics: it couples low-cost in-home sensing, explicit data schemas, privacy-preserving campaign operation, and diagnosis-oriented analysis to produce empirical evidence about building thermal characteristics, heating system behavior, and occupant comfort needs before the heating transition (Hofte et al., 8 Sep 2025).