---
title: AirQo Data Pipeline
url: https://www.emergentmind.com/topics/airqo-data-pipeline
type: topic
---

# AirQo Data Pipeline

Searching arXiv for recent and foundational papers on AirQo and related air-quality data pipelines.
arxiv_search(query="AirQo data pipeline air quality monitoring low-resource settings", max_results=10)
The AirQo Data Pipeline is the production data infrastructure behind AirQo’s AI-enabled air quality monitoring platform. It is described as a modular, cloud-native Extract-Transform-Load system that supports both real-time and batch processing of heterogeneous air quality data across urban deployments in Africa, with the explicit purpose of ingesting, cleaning, aligning, calibrating, storing, and distributing environmental observations at scale under low-resource conditions characterized by intermittent internet, unreliable power, low-bandwidth GSM links, missing data, and limited local compute [2508.14451].

## 1. Definition, scope, and design problem

AirQo’s pipeline was designed to turn raw readings from low-cost air sensors and reference monitors, together with weather data from external APIs, into trustworthy data products for dashboards, APIs, forecasting, and analytics. Its monitoring network spans **400+ devices** across several African cities, and the system is intended to support real-time dashboards, historical analysis, sensor calibration, forecasting, and operational monitoring. The motivating problem is not only scale, but heterogeneity: sources differ in format, update frequency, reliability, spatial coverage, and data quality [2508.14451].

The platform is therefore framed not as a single ingestion script, but as a complete data infrastructure. The paper explicitly characterizes the design as **modular, scalable, fault-tolerant, and cloud-native**, and emphasizes that a conventional ETL implementation is insufficient in environments where connectivity and power are unstable. A central implication is that data engineering, rather than only model selection, is treated as the primary precondition for usable air-quality intelligence.

A common misconception is to reduce the AirQo Data Pipeline to an internal data-cleaning layer for low-cost sensors alone. In the paper’s formulation, however, it is a full ETL/ELT backbone for heterogeneous environmental data, including reference-grade monitors, third-party networks, weather services, warehousing, messaging, model-serving, dashboards, and operations tooling. The system’s scope is therefore infrastructural rather than merely algorithmic.

## 2. Data sources and layered system architecture

The data sources layer includes **AirQo low-cost sensors**, **Reference-grade BAM monitors**, **third-party sensor networks** such as **IQAir, MetOne, and Clarity**, **weather APIs** such as **OpenWeatherMap and TAHMO**, and field devices reporting through **ThingSpeak**. These feeds are normalized downstream into common schemas after source-specific orchestration and cleaning [2508.14451].

The architecture is organized into layers rather than a monolithic processing engine. The orchestration layer is built with **Apache Airflow**, where each workflow is represented as a **Directed Acyclic Graph (DAG)**. The streaming and messaging layer uses **Apache Kafka** as the communication backbone between ingestion jobs and downstream consumers. **Google BigQuery** acts as the central analytical warehouse for raw data, hourly and daily aggregates, merged and calibrated datasets, and forecast outputs. Stateless microservices consume from Kafka or BigQuery, including the **Device Registry Service**, **Event Service**, **Metadata Service**, **Data Management Service**, **Calibrate Service**, and **Predict Service**. Outputs are exposed through the **AirQo API**, public dashboards, **Apache Superset**, and a custom analytics service.

| Layer | Principal components | Function |
|---|---|---|
| Data sources | AirQo sensors, BAM monitors, IQAir, MetOne, Clarity, TAHMO, OpenWeatherMap, ThingSpeak | Environmental data acquisition |
| Orchestration and messaging | Apache Airflow, Apache Kafka | Workflow control, buffering, decoupling |
| Storage and services | Google BigQuery, Device Registry Service, Calibrate Service, Predict Service | Warehousing, calibration, analytics services |
| Infrastructure and access | Kubernetes/GCP, Argo CD, GitHub Actions, Prometheus, Grafana, NGINX, HAProxy, Calico, Google Secret Manager | Deployment, observability, traffic, security |

Airflow’s internal components are explicitly enumerated: **Scheduler**, **Celery workers**, **Metadata database**, and **Webserver**. On the infrastructure side, the system runs on **Google Cloud Platform** with **Kubernetes** clusters for production and staging, **Argo CD** for GitOps deployment, **GitHub Actions** for CI/CD, **Prometheus + Grafana** for monitoring, **NGINX ingress** and **HAProxy** for traffic handling, and **Google Secret Manager** for credentials [2508.14451].

This layered decomposition is significant because it separates orchestration, transport, storage, model services, and presentation. A plausible implication is that AirQo’s design prioritizes operational substitutability: each layer can evolve without requiring a full redesign of the rest of the stack.

## 3. ETL/ELT workflow and DAG semantics

The pipeline is described as a collection of Airflow DAGs implementing near-real-time, hourly, and historical workflows rather than a single monolithic ETL job. In the near-real-time low-cost sensor workflow, the task graph is made explicit through a named sequence:

1. `extract_raw_data`  
2. `clean_data_raw_data`  
3. `save_test_data`  
4. `aggregate`  
5. `extract_hourly_weather_data`  
6. `merge_data`  
7. `calibrate`  
8. `send_hourly_measurements_to_api`  
9. `send_hourly_measurements_to_message_broker`  
10. `send_hourly_measurements_to_BigQuery`  
11. `update_latest_data_topic` [2508.14451]

These tasks formalize the ETL/ELT logic. Raw data are pulled from device APIs, partner APIs, weather APIs, and historical or backfill sources. Before downstream use, readings are checked for type consistency, filtered for outliers, structured for downstream processing, and prepared for resampling. The raw readings are then resampled into **hourly data**, which the paper identifies as useful for normalizing irregular sensor reporting, reducing noise, aligning with weather measurements, and simplifying calibration and forecasting.

Feature enrichment occurs through the merge of hourly air-quality data with hourly weather measurements, site metadata, and other contextual information. The enriched records are then passed to a machine-learning calibration step, after which the outputs are distributed simultaneously to the AirQo API, Kafka topics, BigQuery, and potentially other microservices. BigQuery stores multiple curated forms of the data—**raw**, **averaged**, **consolidated**, and **forecast**—which subsequently support trend analysis, public dashboards, anomaly detection, forecasting, and compliance or reporting [2508.14451].

This design has two important consequences. First, it makes the operational dependencies transparent: extraction precedes cleaning, cleaning precedes aggregation, aggregation precedes weather alignment, and calibration follows feature merging. Second, it separates immediate dissemination from analytical storage, allowing the same calibrated data products to feed both low-latency applications and warehouse-centric analytics.

## 4. Calibration, harmonization, and analytical data products

Calibration is a central functional stage because low-cost sensors drift and are affected by temperature, humidity, and cross-sensitivity. The paper describes a **Calibrate Service** that uses reference-grade monitors and weather features and operates on both streaming and historical datasets. Conceptually, the sequence is: raw low-cost sensor data are collected, weather data are merged in, a calibration model predicts corrected pollutant values, and calibrated outputs are stored and served [2508.14451].

The paper does not provide detailed calibration model equations. Instead, it provides operational metrics that quantify pipeline performance:

$$
\text{Availability Rate} = \left(\frac{\text{Hours with Data}}{\text{Total Hours}}\right)\times 100
$$

$$
\text{Calibration Rate} = \left(\frac{\text{Hours with Calibrated Data}}{\text{Hours with Raw Data}}\right)\times 100
$$

These are explicitly not the calibration model itself. A recurrent misunderstanding is to read the reported **>99.9\%** calibration success as a statement about pollutant estimation accuracy; in the paper, it is an operational statement about coverage and successful execution of the calibration stage. The distinction matters because the pipeline paper emphasizes system reliability and deployment robustness more than model-form disclosure.

The paper also notes that weather features such as temperature and humidity are known to improve calibration and forecasting. This embeds AirQo in a broader class of environmental pipelines in which calibration is not a post hoc correction but a routine production-stage transformation. By distributing the calibrated outputs to both streaming and warehouse layers, the system treats calibration as a first-class data product rather than an offline research exercise.

## 5. Scalability, resilience, and operational behavior

Several design choices are explicitly aimed at resilience in low-resource settings. **Modular DAGs** support reuse, easier debugging, independent updates, and fault isolation. **Kafka-based decoupling** prevents slow consumers from blocking ingestion and permits buffering during outages. **CeleryExecutor** enables distributed parallel task execution. **BigQuery** and **GCP** provide elastic storage and compute without on-premises database management. **Containerization and Kubernetes** support reproducibility and horizontal scaling. **Backfilling** recovers missing periods after outages. **Prometheus**, **Grafana**, logging, and Slack alerts improve mean time to recovery. The lessons section additionally notes **Redis-like in-memory caching** for metadata and calibration models to reduce latency and repeated queries, together with **RBAC**, service accounts, authenticated Kafka access, and **Google Secret Manager** for security and access control [2508.14451].

The reported operational metrics make these claims concrete. The pipeline processes roughly **5.76 million raw air quality data entries per month**, based on approximately 20 data points per hour per device, around 400 active sensors, and roughly 720 hours per month. For March–June 2025, AirQo devices averaged **above 70% availability**, with example values of about **72.81% in April** and **72.08% in June**. External networks were less stable: **IQAir** declined from **64.5% in April** to **39.3% in June**, while **MetOne** dropped from **71.81% in April** to **58.47% in May** due to decommissioning. Calibration success was reported as **99.99% in March**, **99.94% in April**, **99.84% in May**, and **99.99% in June**. The paper also reports near **1:1 parity** between raw aggregated and calibrated hourly data, with over **150k measurements per month** in both categories during several months [2508.14451].

Infrastructure monitoring adds another layer of operational detail. From June 2025 logs, the controller node remained lightly loaded, worker nodes handled most computation, and **Worker-3** showed the highest CPU and memory usage, at times nearing saturation. Network traffic was asymmetric across nodes: the controller had consistent inbound traffic, Worker-1 showed bursty spikes, Worker-2 had relatively high outgoing traffic, and Worker-3 had steady inbound load. These results do not invalidate the architecture; rather, they show that a scalable design still requires ongoing tuning for task balance and resource allocation.

## 6. Position within air-quality pipeline research

AirQo belongs to a broader lineage of air-quality data systems, but it occupies a distinct position within that landscape. The earlier **hackAIR** platform was an end-to-end multimodal air-quality pipeline that combined official open sources, social media, mobile-app images, webcams, and citizen-built sensors, storing heterogeneous observations in a **Sensor Observation Service (SOS)** repository and planning later fusion through **geostatistics**, **residual kriging**, and **land-use regression** [1610.01209]. AirQo differs in emphasis: its core contribution is a production-grade, cloud-native ETL/ELT infrastructure for continuous large-scale operations rather than multimodal visual estimation or a conceptually proposed unified indicator.

The **AQEyes** system addressed a different problem: end-to-end anomaly detection and examination for air-quality station data. Its pipeline handled missing and differently granularized data, augmented pollutant series with weather covariates, trained per-pollutant **LSTM** predictors, and applied dynamic thresholding to prediction errors, with interactive visualization for analyst inspection [2103.12910]. Relative to AQEyes, AirQo is less centered on an analyst-facing visual anomaly workflow and more centered on stable ingestion, calibration, warehousing, and distribution. This suggests that AirQo operates closer to the infrastructural substrate on which systems like AQEyes could be deployed.

The **AQUAIR** dataset extends an “AirQo-style” environmental data pipeline into a smart aquaculture setting, turning continuous low-cost indoor air measurements from a trout hatchery into a quality-controlled, analysis-ready time series. Its preprocessing logic—timestamp normalization, fixed-grid alignment, duplicate-slot averaging, short-gap interpolation, Hampel-based outlier treatment, range enforcement, and standardized CSV export—illustrates how the AirQo design pattern can be adapted outside urban ambient monitoring [2509.24069]. In that sense, AirQo is not only a deployment platform but also a transferable pipeline template.

Taken together, these comparisons clarify what the AirQo Data Pipeline is and is not. It is not merely a calibration script, not only a streaming bus, and not a fully specified unified air-quality index model. It is a cloud-native, modular environmental data backbone whose central achievement lies in making heterogeneous, intermittently available air-quality observations operationally usable at scale in low-resource settings.

Source: https://www.emergentmind.com/topics/airqo-data-pipeline