The IoT message arrived: why is the observation missing? Reliable data delivery

Which delivery boundary does the acknowledgment describe?

Arrival at a gateway, acceptance by a broker, processing by a consumer and durable database commit are different events. A transport acknowledgment describes a particular boundary. If the broker accepts a message but a consumer stops before persistence, the application may still have no observation. The design specifies when the sender may delete its retained copy. A successful send call alone cannot establish that a durable, reviewable record exists centrally.

Required delivery semantics follow the decision. Losing some intermediate samples of a frequently refreshed status may be acceptable, while the same loss can matter for incident evidence or period totals. Classify observations rather than enabling the strongest mechanism indiscriminately. Stronger delivery has cost and latency implications, but critical records cannot rely on an undefined boundary. The contract includes how missing evidence is detected and who investigates it, not merely the selected protocol name.

The observation envelope carries identity and meaning

An observation needs source association, event identity, event time, quantity, units, quality and schema version. A resetting sequence counter requires a boot epoch or another collision-resistant identity scheme. Asset identity remains separate from device identity so replacement hardware does not corrupt historical mapping. Arrival time is additional metadata, not a substitute for acquisition time. Late observations remain late observations, and uncertain source clocks retain an explicit quality indication.

A device field in a payload is not authenticated identity. The server uses trusted connection or application authorization context and checks claims against it. If a gateway represents many legacy devices, document whether authentication actually reaches each physical sensor. Valid JSON also does not establish measurement correctness. Check ranges, units and quality, and route unsupported schemas through an explicit failure path instead of silently interpreting them with a different meaning.

MQTT QoS does not automatically provide exactly-once business processing

MQTT defines delivery levels: QoS 0 permits loss, QoS 1 can redeliver and QoS 2 provides exactly-once delivery within its protocol exchange. Database transactions and downstream business effects remain separate. If a consumer commits a record and then loses its own processing state, a retry must not repeat the business effect. Application-level idempotency can therefore remain necessary even when strong transport delivery is selected.

Session retention, message expiry and retained state serve different purposes. Giving a new subscriber the last known state can help, but it must not appear to be a fresh measurement. Reconnection should not accidentally execute an obsolete command. Commands need issue time, expiry and operation identity. Dropping historical observations and rejecting expired commands are different policies. Transport settings follow data classes rather than extending one convenient default across every workflow.

Section references: OASIS · MQTT 5.0 ↗

A committed write with a lost receipt is an expected failure path

A connection can fail after central persistence but before its receipt arrives. The sender cannot know the write succeeded and retries with the same event identity. The receiver should recognize equivalent content without creating a second business effect. Different content under the same identity is a conflict requiring investigation, not an ordinary duplicate. Silently overwriting the earlier record changes history. Deduplication retention must also cover the possible replay window rather than expire before delayed records can return.

The in-memory model below tests failure before commit, receipt loss after commit, replay, conflicting content and source mismatch. A Map is not durable storage: its write represents a completed persistent transaction. Production needs unique constraints, atomicity and a defined storage durability contract. The example implements neither MQTT nor TLS. Its receipt acknowledges one event identity; acknowledging only the largest sequence number can incorrectly discard unreceived gaps unless contiguous progress has been established.

const assert = require('node:assert/strict');
// In-memory failure model. A real implementation needs durable transactions.
const committed = new Map();
function ingest(source, record, failAt = null) {
  assert.equal(record.tenant, source.tenant);
  assert.equal(record.device, source.device);
  assert(typeof record.epoch === 'string' && record.epoch.length > 0);
  assert(Number.isSafeInteger(record.seq) && record.seq >= 0);
  assert(Number.isFinite(record.value));
  assert.equal(record.unit, 'degC');
  assert.equal(record.schemaVersion, 1);
  assert(['valid', 'suspect', 'invalid'].includes(record.quality));
  assert(Number.isFinite(Date.parse(record.eventTime)));
  const key = JSON.stringify([source.tenant, source.device,
    record.epoch, record.seq]);
  const content = JSON.stringify([record.schemaVersion, record.eventTime,
    record.value, record.unit, record.quality]);
  if (committed.has(key)) {
    assert.equal(committed.get(key), content, 'Conflicting event identity');
    return { key, status: 'persisted', duplicate: true };
  }
  if (failAt === 'before-commit') throw new Error('Write failed');
  committed.set(key, content); // Models a committed durable write, not a real one.
  if (failAt === 'after-commit') throw new Error('Receipt lost');
  return { key, status: 'persisted', duplicate: false };
}
// source must come from a trusted authentication/authorization adapter.
const source = { tenant: 'example-org', device: 'sensor-01' };
const record = { ...source, epoch: 'example-boot-a', seq: 7,
  eventTime: '2026-10-20T08:00:00Z', value: 24.5, unit: 'degC',
  schemaVersion: 1, quality: 'valid' };
assert.throws(() => ingest(source, record, 'before-commit'));
assert.equal(committed.size, 0);
assert.throws(() => ingest(source, record, 'after-commit'));
assert.equal(committed.size, 1);
assert.equal(ingest(source, record).duplicate, true);
assert.equal(committed.size, 1);
assert.throws(() => ingest(source, { ...record, value: 99 }));
assert.throws(() => ingest(source, { ...record, device: 'sensor-02' }));
assert.throws(() => ingest(source, { ...record, seq: -1 }));
assert.throws(() => ingest(source, { ...record, schemaVersion: 2 }));
assert.throws(() => ingest(source, { ...record, quality: 'unknown' }));
assert.equal(ingest(source, { ...record, epoch: 'example-boot-b', seq: 0 }).duplicate, false);
assert.equal(committed.size, 2);
console.log('Commit, lost receipt, replay, conflict and source checks pass');

Outage buffering and reconnection flow control

Define durable retention during an outage, full-storage behavior and visible loss reporting. Live state and historical evidence may need different priorities. Simultaneous fleet reconnection can overload brokers and consumers. Retry delays, jitter, bounded concurrency and backpressure must be considered together. If application receipts slow down, sources should not produce unlimited outstanding work. End-to-end queues need observability rather than merely confirming that each network connection remains established.

Suppose 60 hypothetical devices each produce a 250-byte application record every ten seconds. Aggregate payload is 1,500 bytes/s or 129.6 decimal MB/day. That excludes framing, handshakes, encryption, retries and storage indexes and is not a cellular-plan estimate. Batching changes latency and retransmission volume. Large batches may conflict with short alert deadlines. Capacity is assessed using actual message distributions and outages, not average payload volume alone.

Which part of the data path does encryption protect?

TLS with proper verification protects a transport connection; data exposed at a gateway or broker have a separate lifecycle. Server-only authentication still requires a reliable client-authentication method. Mutual TLS also needs a mapping from the verified credential to application permissions. Channel protection cannot independently prove sensor accuracy or that a trusted intermediary preserved content. If end-to-end content verification is required, define the signing principal, key and exact fields under that separate design.

Private keys and long-lived access tokens do not belong in message payloads, error logs or public documentation. Production identities are separate from development devices. A gateway representing several sensors needs bounded permissions and asset mappings rather than implicit authority over an organization. Human sessions are distinct from device access. A reliable transport can still place a record in the wrong site if identity context is wrong, so security and the observation model must be designed together.

Section references: RFC 8446 · TLS 1.3 ↗

What transport should and should not be expected to provide

Expect traceable delivery, visible gaps and bounded replay under defined conditions, not zero loss, zero delay or unlimited offline retention over every network. Local control and central collection can have different deadlines. Transport choices do not replace physical protection. State whether latency runs from acquisition to reception, persistence or user visibility. These durations answer different questions and should not be combined into one convenient real-time claim.

A connected indicator does not establish fresh data. Monitor queue age, last valid event, expected and accepted counts, duplicates and conflicts separately. Distinguish silent loss from intentional policy drops. Preserve raw quality indicators even when producing a cleaned analytical series. Teams can then ask whether observations are sufficient and correctly contextualized for a decision, rather than merely whether traffic exists. Transport enables that investigation but does not supply the physical meaning itself.

Pilot tests should include lost receipts and source mismatch

Interrupt different stages, restart consumers and replay events while independently counting expected durable records. Inject conflicting content, wrong tenants, stale observations and sequence gaps. Test the actual authentication adapter separately: replacing the trusted source object with a client-controlled JSON field would provide no authentication. Revocation also needs tests against existing sessions, not only new connections. The third article examines this identity lifecycle in more detail.

Gokbilge Tech's telemetry, communications and observability capabilities can address these boundaries within a defined project. Kugrul provides a pilot product context, not a claim that every illustrated mechanism is already delivered. Assessment starts with data classes, outage expectations, identity boundaries and persistence requirements. Success concerns reviewable evidence for the intended decision under stated conditions, beyond establishing an MQTT connection. Private field architectures remain outside the public examples.

• Are the receipt boundary and local deletion condition explicit?
• Is event identity unique across restarts and the replay window?
• Does replay after receipt loss avoid a second business effect?
• Are payload source claims checked against trusted identity context?