> ## Content Index
> Fetch the complete content index at: https://devinside.cloud/llms.txt
> Use this file to discover other available public pages before exploring further.

# Accepted is not completed.
- URL: https://devinside.cloud/an-observable-net10-service-before-the-first-incident/
- Published: 2026-11-12T15:49:29.000Z
- Updated: 2026-11-13T00:58:13.000Z
- Description: The HTTP request took a few milliseconds. The queue accepted a message. The worker produced a log line. The customer still has no result.
- Author: Marcin Borecki
- Tags: Engineering, Data & Observability, #m14, #devinside-review

Historical edition · Sources and examples scoped to 2025-12-10.

*Retrospective technical scope: 10 December 2025\. The order-processing service below is a design example. The interactive traces are calculated models, not captured production traces.*

The HTTP request took a few milliseconds. The queue accepted a message. The worker produced a log line. The customer still has no result.

Every component has a plausible success story. None of those stories explains the operation the customer asked the system to perform. That is the observability gap I would address before choosing a dashboard or arguing about how many events to retain.

The example in this article accepts an order-processing job through an ASP.NET Core endpoint. The endpoint records acceptance, places work on a queue and returns an operation identity. A worker later calls a dependency. The dependency can fail once and succeed on retry. The operation reaches a final state only when the result is durably known to the application.

That small scenario is enough to separate request duration, queue residence, attempt duration and final completion. It also exposes a common mistake: assuming that an asynchronous boundary carries diagnostic context merely because both sides use the same logging library.

My position is that a service is observable when its operators can distinguish the states that lead to different next actions. Collecting more events can help, but only after those distinctions exist. A graph full of spans cannot repair an operation identity that was never recorded.

Try the diagnostic boundary first. Keep all eight operations selected, then turn off queue-context propagation. The worker spans still exist, but the count of complete causal chains changes. Next turn off worker instrumentation and compare the different missing evidence.

## Begin with the customer's verb

The customer says “process this order.” The endpoint says “accepted.” A dependency says “request succeeded.” These verbs may describe different facts. The service needs a stable vocabulary that does not turn the first successful transport operation into the final business outcome.

For the worked example, acceptance means the application has taken responsibility for the job under its declared storage and delivery contract. It does not mean the dependency has completed. An attempt is one execution of the dependency work. Completion means the operation has reached the terminal result the application exposes to the caller.

The diagnostic model should preserve those distinctions. An operation can be pending after a successful HTTP response. An attempt can fail while the operation later succeeds. A retry can increase elapsed time without creating a second user-requested operation. A cancellation request can stop a caller's wait without proving that a committed dependency effect disappeared.

![One operation, two attempts, one final outcome](https://devinside.cloud/content/images/2026/11/devinside/v4/M14/operation-attempts.svg)

[Inspect the full diagram ↗](https://devinside.cloud/content/images/2026/11/devinside/v4/M14/operation-attempts.svg)Swipe the drawing to inspect it.

The drawing gives the operation one identity and each attempt its own place beneath it. The retry belongs to the original request, but it does not overwrite the first failure. If the only retained field is a final `success` Boolean, the service can answer “did it finish?” while being unable to answer “why did it take so long?”

That does not imply that every detail belongs in one gigantic trace. A durable operation record, a metric and a sampled trace have different jobs. The point is to make their relationship intentional rather than to force one instrument to act as all three.

## Trace identity is not business identity

A trace identifier connects diagnostic work. An operation identifier connects the application request and its outcome. Sometimes there is a convenient one-to-one relationship. The design should not depend on that relationship surviving every retry, replay, delayed message or diagnostic policy change.

The operation record therefore keeps an application-owned identifier that is safe to return to the caller and use for a status lookup. Diagnostic spans can carry that identity when its sensitivity and cardinality are appropriate for the destination. Metrics generally need bounded dimensions rather than an unbounded label for every operation.

This separation becomes useful when tracing is sampled out. The application should still know whether a job is accepted or completed. A missing trace cannot mean that the job never existed. Conversely, a visible trace cannot replace the durable receipt that proves a business effect.

There is also a trust boundary. A caller-provided correlation value is not permission to read another operation. It can help join observations after validation, but status authorization still comes from the application's identity and scope rules. Trace continuity is not an access-control mechanism.

## Create spans around decisions, not around every line

.NET 10 was generally available by this article's context date. The instrumentation shape here uses `ActivitySource`, `Activity`, `ActivityContext` and the existing OpenTelemetry .NET 1.9.0 API family. Exact examples are intentionally small; the application queue and envelope types are owned by this design. [.NET 10 announcement](https://devblogs.microsoft.com/dotnet/announcing-dotnet-10/?ref=devinside.cloud).

Start by naming one source for the work the application understands:

```csharp
private static readonly ActivitySource WorkSource =
    new("DevInside.OrderProcessing", "1.0");

using Activity? activity = WorkSource.StartActivity(
    "order.enqueue", ActivityKind.Producer);

activity?.SetTag("work.kind", "order");
activity?.SetTag("work.operation_id", operationId);

```

`StartActivity` can return `null` when the source has no interested listener or the sampling result does not create an activity. The null-conditional calls are therefore part of the instrumentation contract. Expensive diagnostic payload construction should not be performed merely because a nullable activity variable exists in the code. The tagged .NET 10 source exposes these creation and listener decisions. [ActivitySource at v10.0.0](https://raw.githubusercontent.com/dotnet/runtime/v10.0.0/src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/ActivitySource.cs?ref=devinside.cloud).

The span name is a stable operation kind, not a string containing a unique order number. The operation identity is a separate tag. Whether that tag should be exported at all depends on data handling and diagnostic needs; the example does not require putting customer text or arbitrary message bodies into telemetry.

A span should also have an end condition. Does `order.enqueue` end when bytes are handed to a client library, when a broker acknowledges them, or when the application records its responsibility? A useful name cannot compensate for an ambiguous interval. State the boundary in the instrumentation design and preserve it when comparing durations.

## Cross the queue with explicit context

Within a synchronous or ordinary asynchronous call chain, ambient activity context can make parentage convenient. A durable queue message is not a continuation of a local stack. It can be read later, by another process, after the original request and its scopes have ended.

The message therefore needs an explicit propagation contract. W3C Trace Context defines `traceparent` and `tracestate` for interoperable context representation. It includes a sampled flag, but that flag is not a guarantee that every downstream observation will be stored forever. [W3C Trace Context Recommendation, 23 November 2021](https://www.w3.org/TR/2021/REC-trace-context-1-20211123/?ref=devinside.cloud).

In the simplified application envelope, the producer stores context separately from the business payload. A real transport adapter should use the appropriate propagator and validate incoming values rather than treating arbitrary user text as trusted activity state.

```csharp
public sealed record WorkEnvelope(
    string OperationId,
    string? TraceParent,
    string? TraceState,
    string PayloadReference);

Activity? current = Activity.Current;
var message = new WorkEnvelope(
    operationId,
    current?.Id,
    current?.TraceStateString,
    payloadReference);

```

This fragment assumes the application has selected W3C activity IDs. It is an explanatory envelope, not a complete broker client. The payload is represented by an application reference because copying the entire business document into diagnostic headers would mix unrelated contracts.

On the consumer side, the parent context is parsed and used explicitly:

```csharp
ActivityContext parent;
bool hasParent = ActivityContext.TryParse(
    message.TraceParent, message.TraceState,
    isRemote: true, out parent);

using Activity? consume = hasParent
    ? WorkSource.StartActivity(
        "order.process", ActivityKind.Consumer, parent)
    : WorkSource.StartActivity(
        "order.process", ActivityKind.Consumer);

```

The worker loop in this example begins message processing without an ambient activity. A hosting adapter must preserve that assumption or deliberately choose how an existing ambient parent participates.

If parsing fails, the application needs a deliberate fallback. Creating a new diagnostic root can preserve the worker's observations, but it does not restore the missing causal connection. A counter for invalid or missing propagation can make the failure visible. The business operation identity can still support a separate lookup, provided it was carried correctly.

![Two visible traces can still miss one relationship](https://devinside.cloud/content/images/2026/11/devinside/v4/M14/propagation.svg)

[Inspect the full diagram ↗](https://devinside.cloud/content/images/2026/11/devinside/v4/M14/propagation.svg)Swipe the drawing to inspect it.

The broken-edge view is important. A dashboard can show two healthy traces while hiding the fact that they belong to one customer operation. Increasing retention will keep both fragments longer; it will not create the relationship that the envelope omitted.

## Sampling changes which questions a trace can answer

Sampling is a resource policy. It determines which diagnostic observations are retained or recorded under a selected rule. It does not determine whether the application work happened.

For the simple model here, a root decision is made for each operation, and child spans follow the parent decision. OpenTelemetry's tagged 1.9.0 `ParentBasedSampler` delegates root decisions to a configured sampler and distinguishes sampled and unsampled local or remote parents. This article uses that versioned behavior as the baseline, rather than assuming that the latest documentation describes every older implementation. [ParentBasedSampler at core-1.9.0](https://raw.githubusercontent.com/open-telemetry/opentelemetry-dotnet/core-1.9.0/src/OpenTelemetry/Trace/ParentBasedSampler.cs?ref=devinside.cloud).

A sampling fraction does not imply that the particular incident will be visible. Nor does retaining a trace imply that every stage has instrumentation. Sampling and coverage are independent dimensions. A fully sampled request can still disappear at an uninstrumented worker, and a perfectly instrumented worker can have its trace omitted by the selected sampling policy.

The opening model lets you change both. It constructs a small set of synthetic operations with two dependency attempts per operation. You can change root sampling, break propagation at the queue, or turn off worker instrumentation. The visible spans and the number of complete causal chains are recalculated from those inputs.

| Eight synthetic operations, two attempts each   | Recorded modeled spans | Complete causal chains |
| ----------------------------------------------- | ---------------------- | ---------------------- |
| Full selection, propagation and instrumentation | 40                     | 8                      |
| Full selection, missing queue context           | 40                     | 0                      |
| Full selection, worker instrumentation off      | 16                     | 0                      |
| No root selected                                | 0                      | 0                      |

These are calculated counts from the opening model, not trace measurements. The identical count of forty in the first two rows hides a completely different relationship between the spans.

Start with every operation selected, propagation enabled and worker instrumentation present. Each operation can show the whole modeled chain. Turn off propagation while leaving every stage instrumented. You still see worker spans, but they belong to disconnected diagnostic roots; the count of complete ingress-to-worker chains drops. Now keep propagation and disable worker instrumentation. The parent context is available, but no worker span exists to display.

Those are different failures with different repairs. The first concerns carrying context; the second concerns creating observations. Neither is fixed by a more colorful timeline.

The sampling control uses deterministic ranks across the synthetic operations so the result is reproducible. It is not an implementation of OpenTelemetry's trace-ID probability sampler, and the counts are not service measurements. Its purpose is to expose the relationship between a selection decision and the evidence left available.

## Preserve the first failed attempt

In the worked operation, the dependency fails once and succeeds on the second attempt. I would create a separate attempt span for each execution and retain the final operation outcome independently. Reusing one mutable span or log record for “the current attempt” risks erasing the failure that explains the extra work.

The first attempt can record a bounded error category. The second can record success. The parent operation can finish successfully while still containing a failed child attempt. This is not an inconsistency. It is the story the operator needs to understand.

Do not infer the business result from a span status alone. An exception can be handled and the operation recovered. A dependency can return a transport success while its payload fails validation. A business rejection can be an expected terminal result rather than a failure of the service. The instrumentation vocabulary should distinguish those meanings rather than assigning one red or green color to everything.

Retries also need an identity policy. Is this a new attempt of the same operation, or a new operation caused by the user? Are two deliveries duplicates of one message? If a replay can repeat an external effect, the trace should not imply that a new span makes it safe. Diagnostics can explain the protocol; they do not provide its idempotency guarantee.

> **Counterexample.** An operation succeeds on its second attempt. A final success metric is correct, and a trace containing one failed attempt is also correct. Erasing the failed attempt makes the explanation weaker without making the outcome more successful.

## Queue residence needs its own clock boundary

An HTTP span can end before the worker starts. The interval between those events is often the interval the customer cares about. If neither side records enough information to describe it, the trace may make the system look fast precisely while its queue is growing.

The application can record acceptance and completion times using a declared clock strategy. It can also observe queue age or scheduling delay at the appropriate boundary. Comparing timestamps from different machines requires care about clock synchronization and the meaning of each timestamp. A line drawn between spans is not by itself a calibrated measurement of every intermediate delay.

![Give each instrument a question it can answer](https://devinside.cloud/content/images/2026/11/devinside/v4/M14/three-instruments.svg)

[Inspect the full diagram ↗](https://devinside.cloud/content/images/2026/11/devinside/v4/M14/three-instruments.svg)Swipe the drawing to inspect it.

The three instruments in the drawing answer different questions. A durable record answers whether the requested work is accepted or complete. A metric summarizes a bounded dimension across operations. A trace explains the selected execution path. A sampled trace stream is a poor sole source for an exact backlog count; a single status row is a poor explanation of a retry; an average duration is a poor substitute for the selected failing path.

This division also limits the cost of observability. Put stable aggregate questions into bounded metrics. Keep operation-level state where its correctness belongs. Use traces for relationships and detailed paths under a declared policy. Do not serialize the entire world into every log event in the hope that a future query will reconstruct a missing model.

## Test the questions before the incident

The useful readiness exercise is not “Can the dashboard connect?” It is a small set of failures that demand different interpretations.

Accept a job and prevent the worker from receiving it. The application should show pending work rather than completed work. Deliver the message with its trace context missing. The worker may still process it, but the diagnostic discontinuity should be detectable. Fail the first dependency attempt and allow the second to succeed. The final result should remain connected to both attempts. Sample out an operation. Its business state should remain available even when its trace is absent.

These are proposed acceptance scenarios for the design, not reports of tests performed in this historical edition. Their value is that each has an independent expectation. They can reject an attractive dashboard that merely contains events.

I would also test the cost boundary. Instrumentation should not create an unbounded in-process queue when its exporter is slow. Sensitive data should not enter an uncontrolled tag value. A logging exception should not become an alternative business outcome. Those concerns belong to the concrete exporter, transport and application implementation; this small design does not claim to certify them.

Before calling the service observable, ask an engineer who did not write it to explain one deliberately interrupted operation. Give them the operation identity, the documented vocabulary and the available tools. Can they distinguish accepted from completed, a retry from a second request, a sampled-out trace from missing work, and a broken propagation edge from an uninstrumented stage?

If those distinctions are clear, the dashboard has something worth displaying. If they are not, adding more spans is likely to make the ambiguity more expensive.

GO FURTHER

## Inspect the contemporary laboratory.

This is a newly reconstructed historical edition, not a claim of earlier authorship. Later executed experiments, their exact environments and original results are preserved in a separate notebook.

[Open the current laboratory notebook ↗](https://devinside.cloud/lab-notebooks/v4/M14.html)

[Download the current reconstruction sources ↗](https://devinside.cloud/content/files/v4/M14-reconstruction.zip)