AirQo Data Pipeline
- AirQo Data Pipeline is a cloud-native modular ETL system that transforms raw sensor and weather data into actionable air quality insights.
- The system integrates diverse data sources through layered architecture using Apache Airflow, Kafka, and Google BigQuery for real-time and batch processing.
- Designed for low-resource environments, the pipeline emphasizes fault tolerance, scalable calibration, and robust data warehousing for operational analytics.
Searching arXiv for recent and foundational papers on AirQo and related air-quality data pipelines. arxiv_search(query="AirQo data pipeline air quality monitoring low-resource settings", max_results=10) The AirQo Data Pipeline is the production data infrastructure behind AirQo’s AI-enabled air quality monitoring platform. It is described as a modular, cloud-native Extract-Transform-Load system that supports both real-time and batch processing of heterogeneous air quality data across urban deployments in Africa, with the explicit purpose of ingesting, cleaning, aligning, calibrating, storing, and distributing environmental observations at scale under low-resource conditions characterized by intermittent internet, unreliable power, low-bandwidth GSM links, missing data, and limited local compute (Sserujongi et al., 20 Aug 2025).
1. Definition, scope, and design problem
AirQo’s pipeline was designed to turn raw readings from low-cost air sensors and reference monitors, together with weather data from external APIs, into trustworthy data products for dashboards, APIs, forecasting, and analytics. Its monitoring network spans 400+ devices across several African cities, and the system is intended to support real-time dashboards, historical analysis, sensor calibration, forecasting, and operational monitoring. The motivating problem is not only scale, but heterogeneity: sources differ in format, update frequency, reliability, spatial coverage, and data quality (Sserujongi et al., 20 Aug 2025).
The platform is therefore framed not as a single ingestion script, but as a complete data infrastructure. The paper explicitly characterizes the design as modular, scalable, fault-tolerant, and cloud-native, and emphasizes that a conventional ETL implementation is insufficient in environments where connectivity and power are unstable. A central implication is that data engineering, rather than only model selection, is treated as the primary precondition for usable air-quality intelligence.
A common misconception is to reduce the AirQo Data Pipeline to an internal data-cleaning layer for low-cost sensors alone. In the paper’s formulation, however, it is a full ETL/ELT backbone for heterogeneous environmental data, including reference-grade monitors, third-party networks, weather services, warehousing, messaging, model-serving, dashboards, and operations tooling. The system’s scope is therefore infrastructural rather than merely algorithmic.
2. Data sources and layered system architecture
The data sources layer includes AirQo low-cost sensors, Reference-grade BAM monitors, third-party sensor networks such as IQAir, MetOne, and Clarity, weather APIs such as OpenWeatherMap and TAHMO, and field devices reporting through ThingSpeak. These feeds are normalized downstream into common schemas after source-specific orchestration and cleaning (Sserujongi et al., 20 Aug 2025).
The architecture is organized into layers rather than a monolithic processing engine. The orchestration layer is built with Apache Airflow, where each workflow is represented as a Directed Acyclic Graph (DAG). The streaming and messaging layer uses Apache Kafka as the communication backbone between ingestion jobs and downstream consumers. Google BigQuery acts as the central analytical warehouse for raw data, hourly and daily aggregates, merged and calibrated datasets, and forecast outputs. Stateless microservices consume from Kafka or BigQuery, including the Device Registry Service, Event Service, Metadata Service, Data Management Service, Calibrate Service, and Predict Service. Outputs are exposed through the AirQo API, public dashboards, Apache Superset, and a custom analytics service.
| Layer | Principal components | Function |
|---|---|---|
| Data sources | AirQo sensors, BAM monitors, IQAir, MetOne, Clarity, TAHMO, OpenWeatherMap, ThingSpeak | Environmental data acquisition |
| Orchestration and messaging | Apache Airflow, Apache Kafka | Workflow control, buffering, decoupling |
| Storage and services | Google BigQuery, Device Registry Service, Calibrate Service, Predict Service | Warehousing, calibration, analytics services |
| Infrastructure and access | Kubernetes/GCP, Argo CD, GitHub Actions, Prometheus, Grafana, NGINX, HAProxy, Calico, Google Secret Manager | Deployment, observability, traffic, security |
Airflow’s internal components are explicitly enumerated: Scheduler, Celery workers, Metadata database, and Webserver. On the infrastructure side, the system runs on Google Cloud Platform with Kubernetes clusters for production and staging, Argo CD for GitOps deployment, GitHub Actions for CI/CD, Prometheus + Grafana for monitoring, NGINX ingress and HAProxy for traffic handling, and Google Secret Manager for credentials (Sserujongi et al., 20 Aug 2025).
This layered decomposition is significant because it separates orchestration, transport, storage, model services, and presentation. A plausible implication is that AirQo’s design prioritizes operational substitutability: each layer can evolve without requiring a full redesign of the rest of the stack.
3. ETL/ELT workflow and DAG semantics
The pipeline is described as a collection of Airflow DAGs implementing near-real-time, hourly, and historical workflows rather than a single monolithic ETL job. In the near-real-time low-cost sensor workflow, the task graph is made explicit through a named sequence:
extract_raw_dataclean_data_raw_datasave_test_dataaggregateextract_hourly_weather_datamerge_datacalibratesend_hourly_measurements_to_apisend_hourly_measurements_to_message_brokersend_hourly_measurements_to_BigQueryupdate_latest_data_topic(Sserujongi et al., 20 Aug 2025)
These tasks formalize the ETL/ELT logic. Raw data are pulled from device APIs, partner APIs, weather APIs, and historical or backfill sources. Before downstream use, readings are checked for type consistency, filtered for outliers, structured for downstream processing, and prepared for resampling. The raw readings are then resampled into hourly data, which the paper identifies as useful for normalizing irregular sensor reporting, reducing noise, aligning with weather measurements, and simplifying calibration and forecasting.
Feature enrichment occurs through the merge of hourly air-quality data with hourly weather measurements, site metadata, and other contextual information. The enriched records are then passed to a machine-learning calibration step, after which the outputs are distributed simultaneously to the AirQo API, Kafka topics, BigQuery, and potentially other microservices. BigQuery stores multiple curated forms of the data—raw, averaged, consolidated, and forecast—which subsequently support trend analysis, public dashboards, anomaly detection, forecasting, and compliance or reporting (Sserujongi et al., 20 Aug 2025).
This design has two important consequences. First, it makes the operational dependencies transparent: extraction precedes cleaning, cleaning precedes aggregation, aggregation precedes weather alignment, and calibration follows feature merging. Second, it separates immediate dissemination from analytical storage, allowing the same calibrated data products to feed both low-latency applications and warehouse-centric analytics.
4. Calibration, harmonization, and analytical data products
Calibration is a central functional stage because low-cost sensors drift and are affected by temperature, humidity, and cross-sensitivity. The paper describes a Calibrate Service that uses reference-grade monitors and weather features and operates on both streaming and historical datasets. Conceptually, the sequence is: raw low-cost sensor data are collected, weather data are merged in, a calibration model predicts corrected pollutant values, and calibrated outputs are stored and served (Sserujongi et al., 20 Aug 2025).
The paper does not provide detailed calibration model equations. Instead, it provides operational metrics that quantify pipeline performance:
These are explicitly not the calibration model itself. A recurrent misunderstanding is to read the reported >99.9\% calibration success as a statement about pollutant estimation accuracy; in the paper, it is an operational statement about coverage and successful execution of the calibration stage. The distinction matters because the pipeline paper emphasizes system reliability and deployment robustness more than model-form disclosure.
The paper also notes that weather features such as temperature and humidity are known to improve calibration and forecasting. This embeds AirQo in a broader class of environmental pipelines in which calibration is not a post hoc correction but a routine production-stage transformation. By distributing the calibrated outputs to both streaming and warehouse layers, the system treats calibration as a first-class data product rather than an offline research exercise.
5. Scalability, resilience, and operational behavior
Several design choices are explicitly aimed at resilience in low-resource settings. Modular DAGs support reuse, easier debugging, independent updates, and fault isolation. Kafka-based decoupling prevents slow consumers from blocking ingestion and permits buffering during outages. CeleryExecutor enables distributed parallel task execution. BigQuery and GCP provide elastic storage and compute without on-premises database management. Containerization and Kubernetes support reproducibility and horizontal scaling. Backfilling recovers missing periods after outages. Prometheus, Grafana, logging, and Slack alerts improve mean time to recovery. The lessons section additionally notes Redis-like in-memory caching for metadata and calibration models to reduce latency and repeated queries, together with RBAC, service accounts, authenticated Kafka access, and Google Secret Manager for security and access control (Sserujongi et al., 20 Aug 2025).
The reported operational metrics make these claims concrete. The pipeline processes roughly 5.76 million raw air quality data entries per month, based on approximately 20 data points per hour per device, around 400 active sensors, and roughly 720 hours per month. For March–June 2025, AirQo devices averaged above 70% availability, with example values of about 72.81% in April and 72.08% in June. External networks were less stable: IQAir declined from 64.5% in April to 39.3% in June, while MetOne dropped from 71.81% in April to 58.47% in May due to decommissioning. Calibration success was reported as 99.99% in March, 99.94% in April, 99.84% in May, and 99.99% in June. The paper also reports near 1:1 parity between raw aggregated and calibrated hourly data, with over 150k measurements per month in both categories during several months (Sserujongi et al., 20 Aug 2025).
Infrastructure monitoring adds another layer of operational detail. From June 2025 logs, the controller node remained lightly loaded, worker nodes handled most computation, and Worker-3 showed the highest CPU and memory usage, at times nearing saturation. Network traffic was asymmetric across nodes: the controller had consistent inbound traffic, Worker-1 showed bursty spikes, Worker-2 had relatively high outgoing traffic, and Worker-3 had steady inbound load. These results do not invalidate the architecture; rather, they show that a scalable design still requires ongoing tuning for task balance and resource allocation.
6. Position within air-quality pipeline research
AirQo belongs to a broader lineage of air-quality data systems, but it occupies a distinct position within that landscape. The earlier hackAIR platform was an end-to-end multimodal air-quality pipeline that combined official open sources, social media, mobile-app images, webcams, and citizen-built sensors, storing heterogeneous observations in a Sensor Observation Service (SOS) repository and planning later fusion through geostatistics, residual kriging, and land-use regression (Moumtzidou et al., 2016). AirQo differs in emphasis: its core contribution is a production-grade, cloud-native ETL/ELT infrastructure for continuous large-scale operations rather than multimodal visual estimation or a conceptually proposed unified indicator.
The AQEyes system addressed a different problem: end-to-end anomaly detection and examination for air-quality station data. Its pipeline handled missing and differently granularized data, augmented pollutant series with weather covariates, trained per-pollutant LSTM predictors, and applied dynamic thresholding to prediction errors, with interactive visualization for analyst inspection (Liu et al., 2021). Relative to AQEyes, AirQo is less centered on an analyst-facing visual anomaly workflow and more centered on stable ingestion, calibration, warehousing, and distribution. This suggests that AirQo operates closer to the infrastructural substrate on which systems like AQEyes could be deployed.
The AQUAIR dataset extends an “AirQo-style” environmental data pipeline into a smart aquaculture setting, turning continuous low-cost indoor air measurements from a trout hatchery into a quality-controlled, analysis-ready time series. Its preprocessing logic—timestamp normalization, fixed-grid alignment, duplicate-slot averaging, short-gap interpolation, Hampel-based outlier treatment, range enforcement, and standardized CSV export—illustrates how the AirQo design pattern can be adapted outside urban ambient monitoring (Sabiri et al., 28 Sep 2025). In that sense, AirQo is not only a deployment platform but also a transferable pipeline template.
Taken together, these comparisons clarify what the AirQo Data Pipeline is and is not. It is not merely a calibration script, not only a streaming bus, and not a fully specified unified air-quality index model. It is a cloud-native, modular environmental data backbone whose central achievement lies in making heterogeneous, intermittently available air-quality observations operationally usable at scale in low-resource settings.
