Verifying Connection Coalescing with DevTools

Diagnosis: two hostnames that satisfy every documented coalescing precondition — one resolved IP, port 443, the same ALPN token and one certificate whose Subject Alternative Name list covers both — still show two different Connection IDs and two full TLS handshakes in the Network panel, and nothing in the panel tells you which precondition the browser actually rejected.

Root cause: the pool is keyed on more than the four preconditions

Chromium does not look up “a connection to this host”. It looks up a session key, and for HTTP/2 that key is the SpdySessionKey: scheme, host, port, proxy chain, session usage, the network anonymization key, the secure-DNS policy and the privacy mode. A direct lookup on that key only ever finds a session for the identical origin — that is ordinary connection reuse, and it is what you see when two requests for cdn.example.com share a Connection ID. Coalescing is the second lookup: when the keyed lookup misses, SpdySessionPool searches its IP-to-session index for a live session whose peer address appears in the DNS result for the new hostname, and then asks that session whether it may be pooled. That CanPool check re-verifies the presented certificate against the new hostname, checks Certificate Transparency compliance and client-certificate state, and requires the two keys to be otherwise identical. Only when all of that passes does the request join the existing session. HTTP/3 runs the equivalent path over QuicSessionKey, in a pool that is entirely separate from the HTTP/2 one.

Two consequences fall out of that design and they cause most false alarms. First, the pools are partitioned by top-frame site: the network anonymization key is part of the key, so a script loaded from cdn.example.com inside a cross-site iframe is looked up under a different key than the same host requested by the top-level document, and it will open its own connection no matter how perfect the certificate is. Second, an Alt-Svc upgrade splits your traffic: once one hostname is being served over h3 and another is still on h2, they are in different pools by construction, and the two Connection IDs you are staring at are correct. Rolling h3 out per hostname — a common step when rolling out Alt-Svc headers safely — produces exactly this state during the transition.

The third consequence is the one that makes the symptom intermittent. The IP-pool lookup runs when the request starts, against sessions that already exist. If the preload scanner discovers cdn.example.com/app.css and static.example.com/ui.js in the same batch of markup, both connect jobs start within a few milliseconds of each other, both find an empty pool, and both complete — Chromium does not retroactively fold the second session into the first once it notices the overlap. You then keep two sessions for the rest of the page, and every subsequent request from either hostname reuses its own. Coalescing is therefore a race whose outcome depends on RTT, on where in the HTML the second hostname first appears, and on whether a previous navigation left a warm session in the pool. Verification has to be able to tell that race apart from an outright precondition failure, because the fixes are completely different: one is a certificate or DNS change, the other is a markup ordering change or an argument for removing the second hostname altogether, as covered in dismantling domain sharding for HTTP/3.

What the Network panel can and cannot prove

The Network panel is the fastest instrument and the easiest one to misread. Enable Protocol, Remote Address and Connection ID from the column-header context menu; Connection ID on its own answers “did these two rows share a socket” but never “why not”. Two hostnames with the same Connection ID are coalesced. Two hostnames with different Connection IDs are not — but the reason is in the other two columns, and several rows in any real waterfall cannot testify at all.

Three readings in that table are traps. Rows 1 and 2 look like a coalescing success and are nothing of the kind — they are the same origin, so the keyed lookup hits and the IP pool is never consulted. Row 5 tempts you into counting a “reused” row as evidence; a memory-cache hit reports Connection ID 0 and an empty Remote Address because no socket was involved, and the same is true of disk-cache hits, service-worker responses and prefetch-cache hits. Row 6 looks like a failure and is not: assets.example.com negotiated h3 while the others are on h2, and those pools never merge. The one genuine failure is rows 3 and 4, and the Remote Address column names the cause — the anycast answer for static.example.com came from a different edge address, so the IP-pool index has nothing to match. Reading the waterfall’s timing bars instead of these columns is a dead end for this question; the network waterfall’s timing phases hide a connect bar for cached rows exactly as they do for reused ones.

Minimal reproduction

The whole bug fits in one document. Both hostnames resolve through the same CNAME chain, one certificate covers both, and both are discovered by the preload scanner in the same batch:

<!-- Both subresources are discovered by the preload scanner in one pass, so both
     connect jobs start within a few milliseconds. The IP-pool lookup for the
     second host runs while the first session is still handshaking and finds an
     empty pool — the browser cannot join a session that does not exist yet. -->
<link rel="stylesheet" href="https://cdn.example.com/app.7f2c.css">
<script src="https://static.example.com/ui.4b1a.js" defer></script>

Before blaming the race, confirm the preconditions from the shell. These three commands take a minute and rule out the two changes that need a certificate or a DNS edit rather than a markup change:

# 1. Does the browser's resolver see one address, or one per hostname? Query the
#    SAME resolver the browser uses — a coalescing decision is made on the address
#    list of the new host, so two different anycast answers make it impossible.
for h in cdn static assets; do printf '%-8s ' "$h"; dig +short "$h.example.com" | tail -1; done

# 2. Does the certificate the EDGE presents cover every hostname? Ask for the
#    hostname you want to coalesce ONTO, with SNI set to the primary — this is the
#    certificate CanPool will re-verify against the second hostname.
openssl s_client -connect cdn.example.com:443 -servername cdn.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

# 3. Which ALPN token does each hostname actually negotiate? A host that has moved
#    to h3 lives in a different session pool and will never join an h2 session.
for h in cdn static assets; do
  printf '%-8s ' "$h"
  openssl s_client -alpn h2 -connect "$h.example.com:443" </dev/null 2>/dev/null \
    | grep -m1 'ALPN protocol'
done

If all three agree and you still see two Connection IDs, you are looking at the race, and the way to prove it is to remove the race from the experiment. Stage the second hostname’s first request several hundred milliseconds after the first session is known to exist:

// Staged coalescing probe. Scheduling rationale: the IP-pool lookup happens when a
// request STARTS, so issuing the second hostname's request only after the first
// session has been established is the difference between "cannot coalesce" and
// "did not get the chance". Run this in a page whose markup references only the
// first hostname, so the preload scanner cannot start a competing connect job.
const primary = 'https://cdn.example.com/app.7f2c.css';
const probe   = 'https://static.example.com/ui.4b1a.js';

await fetch(primary, { cache: 'no-store' });        // session #1 now exists in the pool
await new Promise((r) => setTimeout(r, 250));       // let the handshake fully settle

await fetch(probe, { cache: 'no-store' });
const [entry] = performance.getEntriesByName(probe);
// A coalesced request performs no lookup and no handshake, so the spec has these
// three attributes return the same value as fetchStart. Non-zero deltas mean a new
// connection was opened and the two hostnames are genuinely not poolable.
console.log({
  dnsMs: entry.domainLookupEnd - entry.domainLookupStart,
  tcpMs: entry.connectEnd - entry.connectStart,
  protocol: entry.nextHopProtocol,   // empty string = timings are TAO-gated, not zero
});

Probe 2 coalescing while probe 1 does not is a complete diagnosis: the certificate, the address and the ALPN token are all fine, and the only defect is that your markup asks for both hostnames before either session exists. Probe 2 also failing moves you back to the precondition list, where the order of checks matters because each one invalidates the next.

Deterministic verification protocol

Work top to bottom. Steps 1 to 3 make the measurement trustworthy; 4 to 7 identify which precondition failed; 8 to 10 turn the answer into something you can prove and keep.

• [ ] 1. Start from a cold socket pool. Open chrome://net-internals/#sockets and click Flush socket pools, or measure in a throwaway profile. A warm pool left by the previous navigation makes every hostname report a reused connection, which is indistinguishable from coalescing in the panel.
• [ ] 2. Disable the cache and unregister service workers for the run. Every row served from a cache or a worker reports Connection ID 0 and contributes nothing. If more than a couple of rows read 0, your measurement is mostly noise.
• [ ] 3. Enable Protocol, Remote Address and Connection ID. Right-click any column header. Connection ID alone tells you whether; the other two columns tell you why not, and you will need both before opening a NetLog.
• [ ] 4. Compare Remote Address, never a local dig. Secure DNS, a corporate resolver, Happy Eyeballs address selection and a multi-record anycast answer can all give the browser a different address from the one your shell sees. The column is authoritative; the shell is a hint.
• [ ] 5. Compare the protocol tokens before anything else. h2 and h3 sessions live in separate pools. If one hostname has been upgraded through Alt-Svc and the other has not, the two Connection IDs are correct and there is nothing to fix except finishing the rollout.
• [ ] 6. Read the certificate the browser received, not the one you issued. Security panel → the origin → View certificate, for each hostname. A stale certificate on one edge node, or a CDN that serves a per-hostname certificate rather than your multi-SAN one, is the most common real SAN gap and is invisible from your own build pipeline.
• [ ] 7. Confirm both requests share a top-frame site. A resource inside a cross-site iframe is keyed under a different network anonymization key and cannot join the top-level page’s session. Load the same URL from the top-level document to confirm the partition is the cause.
• [ ] 8. Take a NetLog and require the positive event. Only HTTP2_SESSION_POOL_FOUND_EXISTING_SESSION_FROM_IP_POOL proves a request joined a session belonging to a different hostname. Its absence, combined with several HTTP2_SESSION_POOL_IMPORTED_SESSION_FROM_SOCKET events, is the machine-readable form of “no coalescing happened”.
• [ ] 9. Run the staged probe to separate a race from a precondition failure. If probe 2 coalesces and probe 1 does not, stop editing DNS: change what the markup asks for first, or drop the second hostname.
• [ ] 10. Cross-check other engines and the field with Resource Timing. Safari and Firefox expose no connection identifier, so verify there with the timing attributes and, at the origin, with a connection-serial log line — see below.

Counting the pool events in a NetLog

The NetLog JSON stores event types as integers and ships the name-to-integer map in its own header, so resolve the constant before counting rather than grepping for the string:

# Record with chrome://net-export → "Stop Logging" BEFORE parsing: a still-running
# capture leaves the events array unterminated and jq will refuse the file.
# Scheduling rationale: FOUND_EXISTING_SESSION_FROM_IP_POOL is emitted at the exact
# moment a request skips its own connect job and adopts another host's session, so
# counting it measures coalescing events rather than the absence of handshakes.
POOL=$(jq -r '.constants.logEventTypes.HTTP2_SESSION_POOL_FOUND_EXISTING_SESSION_FROM_IP_POOL' netlog.json)
NEW=$(jq -r '.constants.logEventTypes.HTTP2_SESSION_POOL_IMPORTED_SESSION_FROM_SOCKET' netlog.json)

jq --argjson p "$POOL" --argjson n "$NEW" '
  { coalesced: [.events[] | select(.type == $p)] | length,
    new_sessions: [.events[] | select(.type == $n)] | length }' netlog.json
# Healthy consolidated origin, three hostnames: { "coalesced": 2, "new_sessions": 1 }

There is no equivalent single event for QUIC. For HTTP/3, filter the log to QUIC_SESSION sources and check that the HTTP_STREAM_JOB entries for both hostnames reference the same source id — same evidence, one extra join.

Engine-neutral proof, from the client and from the origin

Resource Timing gives you the same answer in every browser and in the field, provided you respect one caveat: a cross-origin entry without Timing-Allow-Origin reports zeros for the connection phases, which is trivially mistaken for a coalesced request. nextHopProtocol is the tell — it is an empty string precisely when the timings are opaque.

// Field check. Scheduling rationale: a request that joined an existing session
// performs no lookup and no handshake, so the spec requires domainLookupStart,
// connectStart and connectEnd to all report fetchStart. Any positive delta is a
// connection this page paid for and therefore did not coalesce.
new PerformanceObserver((list) => {
  for (const e of list.getEntries()) {
    if (!e.nextHopProtocol) continue;          // TAO-gated: zeros are opaque, not proof
    const host = new URL(e.name).host;
    const setupMs = Math.round(e.connectEnd - e.domainLookupStart);
    if (setupMs > 0) {
      navigator.sendBeacon('/rum/coalescing', JSON.stringify({
        host, setupMs, protocol: e.nextHopProtocol,
      }));
    }
  }
}).observe({ type: 'resource', buffered: true });

The origin can confirm the same thing without any client cooperation, which is the only practical way to verify Safari at scale. nginx exposes a per-connection serial number, so two hostnames logged against one serial were carried on one connection:

# Scheduling rationale: $connection is the connection serial number and
# $connection_requests counts the requests multiplexed over it. Two different
# $host values sharing one $connection is coalescing, observed at the origin —
# the only signal that survives a client whose DevTools exposes no socket id.
log_format coalescing '$connection $connection_requests $server_protocol '
                      '$host $request_uri $ssl_server_name';
access_log /var/log/nginx/coalescing.log coalescing;

# Then: awk '{print $1, $4}' coalescing.log | sort -u | awk '{c[$1]++} END {for (k in c) print k, c[k]}'
# A serial with 2+ distinct hostnames is a coalesced connection.

Before and after

One catalogue page, three asset hostnames behind one CDN property, measured on a cold profile at 170 ms RTT. “Before” is the state the team believed was already coalescing because the Network panel showed no repeated connect bars; “after” is the same page once static.example.com was re-pointed at the same edge hostname as cdn.example.com and the h3 rollout was completed on all three names.

The row that changed the argument internally was the fourth one. Before the NetLog count, “we see no extra connect bars” was the whole case for coalescing, and it was wrong for a mundane reason: on the repeat views everyone was testing, the pool was warm and every hostname reported reuse. Two FOUND_EXISTING_SESSION_FROM_IP_POOL events on a cold load is a claim that cannot be produced by a warm cache, a service worker or a lucky waterfall. The remaining 0.4% of field entries with non-zero setup are first-visit races on very slow connections — the residue the staged probe predicts and the reason the parent topic on coalescing and sharding treats consolidation of the hostnames themselves as the durable fix.

FAQ

Q: The Connection ID column reads 0 for every row. Is coalescing broken?

No — a 0 means no socket was involved in serving that row, not that the socket was shared. Memory-cache hits, disk-cache hits, responses synthesised by a service worker, prefetch-cache hits and data: URLs all report 0 no matter how the underlying sessions are pooled. This is the single most common misreading of the column, and it usually appears when someone reloads normally instead of hard-reloading. Disable the cache, unregister the service worker for the duration of the measurement, flush the socket pools, and re-run: every row you intend to judge must have been fetched over the network, and every row that was not is silent evidence rather than negative evidence.

Q: DevTools shows one Connection ID but my origin logs two connections. Which one is lying?

Neither. On a navigation where HTTP/3 support has not yet been confirmed for the origin, Chromium starts a QUIC attempt and a TCP attempt in parallel and abandons the loser once the winner completes — the origin sees a connection that never carried a request. A preconnect hint for a host that then gets coalesced produces the same artefact, and so does a cross-site frame with its own partition. Resolve it from the origin side by logging the request count per connection: an abandoned racer or a wasted preconnect shows a serial with zero or one request, while the real session carries the page’s whole asset set. That request-count column is also what tells you whether a preconnect hint is earning its handshake.

Q: Everything matches and NetLog shows the IP-pool event, but only on some loads. Why?

Because coalescing is a lookup performed at request-start time, not a stable property of the two hostnames. When the preload scanner discovers both hostnames in one pass, the second connect job begins while the first is still handshaking, finds nothing in the pool and completes its own session, which the browser then keeps for the rest of the page. The inversion rate follows how early in the HTML the second hostname first appears and how long the first handshake takes, so it rises with RTT and falls on repeat views with a warm pool. Measure it with the staged probe rather than by reloading: if probe 2 coalesces reliably and probe 1 does not, the fix is in your markup ordering or, better, in having one fewer hostname to coalesce. Field-sampling the same signal alongside nextHopProtocol is covered in collecting nextHopProtocol with Resource Timing.

Related

• Dismantling Domain Sharding for HTTP/3 — what to do once verification proves the second hostname is never going to coalesce reliably
• Collecting nextHopProtocol with Resource Timing — turning the same timing attributes into a field signal
• Up: Connection Coalescing & Domain Sharding