A reusable foundation for different IoT data sources
This architecture exploration asks how common IoT data-flow responsibilities can be organized into a reusable starting point. The original design covers collection, ETL, storage, application access, and notifications, while allowing the components to vary with the domain.
The diagram above is the original architecture from my published IoT design article.
Two ingestion modes meet at the data gateway
- Device-driven · Broker or HTTP
- The proposal accepts device/edge-driven data through a broker such as EMQX or Kafka, or through HTTP.
- Host-driven · Trigger or scheduler
- Scheduled collection is represented by triggers such as an Azure Function or schedulers such as Hangfire and Quartz.
- ETL · Extract, transform, load
- The gateway connects to the ETL responsibility, which handles the data-flow work before the application consumes it.
- Storage · Relational and cache options
- The design lists PostgreSQL with TimescaleDB or SQL Server, alongside Redis in the platform's in-memory data responsibilities.
Keep ingestion, internal messaging, and notification concerns distinct
- Data gateway broker
- The external-facing gateway accepts device data and host-driven inputs.
- Internal-purpose broker
- A separate broker connects the ETL, backend, and notification responsibilities. The proposal lists EMQX/MQTT or Kafka as options.
- Application and web tiers
- The API/app tier includes Redis and Lazy Cache. The user-facing tier includes web/mobile applications, Grafana, and Angular or React options.
- Notification hub
- The proposed notification responsibility has its own Redis store, alert rules, and email, SMS, and push-notification channels.
Separating these responsibilities is the central design idea: collection, transformation, application access, and notification processing can be considered independently as the solution evolves.
Published design, implementation scope, and future work
This is a personal architecture and development exploration. The published diagram documents a proposed generic starting point whose final components depend on the target system's requirements.
The original article identifies source IP allowlisting, payload compression, and encryption as future considerations. They remain design considerations in this case study.
For a concrete enterprise monitoring delivery, see CFEMS. For the full background and original component list, read the architecture article.