Must sensor data disappear when the internet goes down?

Why distinguish connectivity loss from measurement loss?

When a remote pump station cannot reach the central service for two hours, it may continue measuring and retain the records, or it may lose the measurements entirely. Both can initially appear as an empty chart. If the architecture cannot distinguish them, the operator's dashboard cannot explain what the system actually knows. The engineering problem is to separate the measurement lifecycle from connection availability, rather than assuming a permanently available network.

Define the guarantee boundary first: at which point is a record considered protected? A driver writing to memory, local storage committing a transaction and the central database accepting an event are different milestones. The numbers here are a hypothetical sizing exercise, not measured Kugrul capacity or a description of its current queue implementation. The proposed acceptance target concerns records durably accepted by the device. Measurements that the sensor never exposes or the device never reads require a separate availability assessment.

How should a measurement's identity and time be modeled?

Using a timestamp alone as the primary key is fragile: clocks can move backward, samples can share a timestamp resolution and devices restart. One possible identity combines a provisioned device identifier, a unique boot or acquisition epoch and a sequence number increasing within that epoch. Verify that epochs really change after restarts and that provisioning does not accidentally assign one device identity to several units. Retries must retain the original event identity rather than inventing a new one on each delivery attempt.

Keep event time separate from central acceptance time. A pressure reading arriving now after a two-hour interruption does not describe a pressure change happening now. OpenTelemetry's log data model distinguishes event and observation timestamps, a useful reference for this design distinction. Consider recording clock synchronization state and uncertainty as well. UTC simplifies time-zone presentation but cannot correct an inaccurate device clock. Charts and alarm rules must state which time they use and what happens when the source timestamp is unreliable.

Section references: OpenTelemetry — Logs Data Model ↗

Why is an in-memory buffer insufficient?

An in-memory queue can bridge a network outage but loses its contents if power also disappears. A durable queue needs a defined meaning for 'written' across the application, operating system and storage device. A successful append call does not necessarily establish survival after power loss. SQLite's WAL documentation gives a concrete example of how commit and synchronization settings affect durability. Merely choosing WAL does not establish lossless behavior on every storage stack; filesystem and device behavior still need verification under the intended conditions.

Committing every sample and committing small batches trade write load, latency and the uncommitted loss window. If a one-second batch remains only in memory, its samples must not be reported as durably accepted. Define queue format versions, partial trailing records, corrupt segments and recovery scanning. When storage fills, report the condition or apply an explicitly agreed reduction policy. Silently deleting the oldest data can make a dashboard appear healthy while breaking the very retention guarantee the queue was introduced to provide.

Section references: SQLite — Write-Ahead Logging ↗

How much storage is needed for a 48-hour outage?

Assume 20 measurement sources each produce one record per second, with an average encoded application record of 160 bytes before compression. Raw backlog is source count × records per second × bytes per record × outage duration. For 48 hours, that is 20 × 1 × 160 × 172,800 = 552,960,000 bytes, or 552.96 decimal MB and approximately 527.34 MiB. These assumptions must be replaced by measurements of representative records, including actual metadata and encoding, before purchasing storage or promising an outage window.

Applying an illustrative factor of 1.8 for local indexes, segment metadata and operating allowance produces 995,328,000 bytes. That factor is a design assumption to validate against actual file growth, not a universal storage constant. A 2 GiB queue allocation leaves headroom in this scenario but excludes space reserved for the operating system and other applications. Repeat the calculation for higher sample rates and larger records. Do not base a retention commitment on an unmeasured compression ratio or assume nominal device capacity equals space available to the queue.

Does MQTT QoS 2 provide application-wide exactly-once processing?

MQTT delivery levels describe exchanges within the protocol. Whether a receiving application commits a database transaction, or loses its response after committing, is a separate transaction boundary. Selecting QoS 2 therefore does not establish exactly-once business processing across an entire measurement pipeline. The OASIS standard defines the protocol behavior. At the application boundary, design an acknowledgement tied to the original event identity and issue it after the receiving system has completed the acceptance action promised by that acknowledgement.

The SQLite example shows how a unique event identity makes repeated insertion harmless: two deliveries leave one row. This covers that insertion only; side effects such as notifying another system need their own design. A repeated identity carrying a different payload is a conflict, not an ordinary retry. Production ingestion should compare a stable payload digest, surface mismatches and define the acknowledgement returned to the device. The example deliberately does not implement those additional responsibilities or claim a complete message-processing system.

CREATE TABLE readings (
  device_id TEXT NOT NULL,
  epoch TEXT NOT NULL,
  seq INTEGER NOT NULL CHECK (seq >= 0),
  event_time TEXT NOT NULL,
  received_time TEXT NOT NULL,
  value REAL NOT NULL,
  PRIMARY KEY (device_id, epoch, seq)
);
BEGIN IMMEDIATE;
INSERT INTO readings VALUES
 ('pump-07', 'boot-a', 81, '2026-10-19T10:00:00Z',
  '2026-10-19T12:00:00Z', 4.2)
ON CONFLICT (device_id, epoch, seq) DO NOTHING;
INSERT INTO readings VALUES
 ('pump-07', 'boot-a', 81, '2026-10-19T10:00:00Z',
  '2026-10-19T12:00:02Z', 4.2)
ON CONFLICT (device_id, epoch, seq) DO NOTHING;
COMMIT;
SELECT COUNT(*) AS stored_readings FROM readings;
-- Expected: 1. Same event delivered twice, stored once.

Section references: OASIS — MQTT 5.0 ↗

How long does backlog recovery take?

After reconnection, new measurements continue while old records are replayed. Assuming measured application goodput of 64,000 bytes per second, subtracting the 3,200-byte live stream leaves 60,800 bytes per second for backlog. The raw 552,960,000-byte backlog then needs approximately 9,094.74 seconds, or 2.53 hours. This idealized calculation assumes sustained goodput and an uninterrupted connection. Transport overhead and retransmission effects must already be reflected in the measured application goodput rather than deducted twice or ignored.

If available goodput is lower than live production, the backlog never drains. Separate policies may be needed for live visibility and historical completeness. A system might prioritize current samples while rate-limiting historical segments, with the interface identifying intervals filled retrospectively. Historical threshold crossings must not silently become fresh real-time alarms. Also test many devices reconnecting together: staggered backoff and central quotas can prevent recovery traffic from overwhelming the service precisely when connectivity returns.

When may a local record be deleted?

A successful send call alone may not justify deletion. In the proposed design, local removal follows durable central acceptance and a matching application acknowledgement. For cumulative acknowledgements, define whether the boundary means a contiguous accepted prefix or merely the largest sequence observed. Deleting through the largest observed sequence while gaps remain can destroy unaccepted data. Segment reclamation must also remain consistent with concurrent sending and recovery after restart; cleanup is part of the data protocol, not just a background housekeeping task.

Monitor queue depth, oldest-record age, free space, last durable acknowledgement and retry rate together. A connected socket with a stationary acceptance counter may indicate a failure beyond transport. If a segment is corrupt, an explicit quarantine and recovery policy can preserve diagnostics while allowing intact segments to proceed. Silently skipping it conceals a loss. Present measurement loss, delivery delay and connection failure as different operating states so a green connectivity indicator cannot mask a queue that has stopped making progress.

Which tests should an offline-retention claim survive?

Generate an independent set of input identities in a controlled test. Interrupt networking, crash the process and remove power at different boundaries, especially before local persistence, after local commit and before the central acknowledgement arrives. After recovery, compare the expected identity set with the stored set. Row counts alone are insufficient because a missing event and a duplicate can cancel numerically. Compare payload digests and expected sequences as well, and state exactly which input boundary the expected set represents.

Report more than a single pass indicator: count missing accepted events, conflicting duplicates, recovery duration and records with uncertain time. Combine full storage with repeated restarts instead of exercising only isolated failures. The checklist below is a proposal for engineering and pilot review, not a field-proven product guarantee. Turn its objectives into commitments only after verification on the intended device, operating system, storage medium, connection profile and representative data volume. Retain the test inputs and results so later firmware changes can be checked against the same baseline.

• Every durably accepted event identity is present centrally with a matching payload digest.
• Repeating an event does not add a row; a different payload produces a visible conflict.
• Queue saturation, power interruption and historical replay remain independently observable.

How does this architecture relate to Kugrul?

Kugrul is publicly positioned around remote data collection and hardware/software integration, with pilot availability. The queue, MQTT and acknowledgement design in this article is an engineering examination, not a statement that these exact algorithms or numerical capacities are already implemented in the product. Bring source counts, sampling requirements, longest expected interruption, power conditions and retention objectives to a technical review. Those inputs help distinguish existing functionality, device-specific adaptation and additional development work.

Remote telemetry engineering addresses the physical collection path; data and observability engineering is relevant to central acceptance, delay and missing coverage. The review should go beyond asking whether something works offline. Ask which records survive which failures, when acceptance occurs and what experiment demonstrates that behavior. In a proposed pilot, consider the source, device, storage and central service together. Make every transfer boundary and its responsible component explicit, then use the related product and capability pages to scope a concrete, testable integration.