Debugging log

It wasn't the 10GbE adapter

Date
Tags
homelabnetworking
Symptom

Transfers to the NAS ran at gigabit speeds over a link the Mac reported as 10Gbase-T, full duplex, active

Cause

The two machines were on different subnets, so every packet went through the gateway instead of staying on the switch — because I had filed the NAS on the management network when I built it

If you do networking for a living you’ll have this diagnosed by the second code block, and the command you’re waiting for me to run is route get. I don’t, and it cost me an evening.

I bought a Thunderbolt 10G adapter to get a Mac mini talking to a UNAS Pro faster than gigabit. I got gigabit.

Everything I knew about these adapters pointed at the adapter: thermal throttling, driver problems, chipset lottery. It was the newest and least trusted thing in the path, and it’s the part you’ll be tempted to return.

It wasn’t the adapter. The Mac and the NAS were on different subnets, so every packet between two machines hanging off the same switch was going up to the gateway to be routed and coming back down again. Putting them on one network took the same benchmark from gigabit to 200–300 MB/s writes and 300–700 MB/s reads, on the same hardware, with the adapter still in it.

What I had to work with, which is less than I’d like

My entire before-measurement is that I said “I’m seeing gigabit speeds.” No figure, no read/write split, and no note on whether the number sat flat or decayed over the length of a transfer. That last one is what I’d want back: a flat ceiling and a decaying one point at different classes of problem, and I threw the distinction away before I started.

The one thing I did check was the link:

❯ ifconfig en18
en18: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=567<RXCSUM,TXCSUM,VLAN_MTU,TSO4,TSO6,AV,CHANNEL_IO>
	inet 192.0.2.10 netmask 0xffffff00 broadcast 192.0.2.255
	media: autoselect (10Gbase-T <full-duplex>)
	status: active

en18 is where my adapter landed; yours will be a different number, and networksetup -listallhardwareports maps hardware to interface names.

10Gbase-T, full duplex, active. That rules out the cheap, common failure — negotiation, where a bad cable or a flaky port drops you to 1G and the readout says so. What it does not rule out is a dirty link: 10GBASE-T over marginal copper will negotiate at 10G, come up active, and then throw errors under load that collapse throughput while this output stays perfect. The split I wanted here needs the interface’s error counters, and I never pulled them.

The evidence turned up while I was testing the wrong theory

The leading theory by this point wasn’t the dongle. It was that the traffic wasn’t on the dongle at all: the mini has a built-in 1G port, and if that was also plugged in, macOS could have been sending storage traffic out the slow one. So I went looking for which interface the traffic used.

route get 198.51.100.10
   route to: nas.example.com
destination: nas.example.com
    gateway: router.example.com
  interface: en18
      flags: <UP,GATEWAY,HOST,DONE,WASCLONED,IFSCOPE,IFREF,GLOBAL>
 recvpipe  sendpipe  ssthresh  rtt,msec    rttvar  hopcount      mtu     expire
       0         0         0         6         8         0      1500         0

interface: en18 answered the question I asked. The answer to the question I hadn’t asked is two lines above it: gateway: is populated with the router and flags: contains GATEWAY, so the destination isn’t on-link. Read that against the inet line from ifconfig and the two addresses sit in different /24s.

The path was never Mac to switch to NAS. It was Mac, switch, gateway, switch, NAS, for every packet.

I should be straight about who spotted that, because it’s the whole lesson. I pasted both blocks into a Claude Code session without comment, having read the one field I went looking for. Claude read the same output and opened its next message with “OH. There it is.” The line that mattered was on my screen the first time I looked at it, and I skipped it because I was checking a different theory. The rest of the session ran the same way — me pulling data, Claude reading it back — which is also how the camera post on this site went.

On Linux, ip route get <destination> prints the same two tells in a different shape.

What this proves, and what it doesn’t

It proves the path is routed. It does not prove the routed path is what was slow.

It also doesn’t kill the theory I ran the command for, and I said it did for longer than I should have. route get is a control-plane lookup: it tells you where a new connection to that address would go, not where an established SMB session already went. A mount that came up while only the 1G port was live stays bound to that interface until something remounts it, and moving both machines onto one network changes an address at one end and does exactly that. So the test that confirmed the diagnosis removed the routed path and re-established the session in the same move, and I never established whether the built-in port was connected in the first place.

Which matters, because of what the two explanations predict. A session pinned to a 1G interface predicts a specific ceiling. A gateway short of forwarding headroom predicts an arbitrary one. All I recorded was “gigabit speeds,” which fits the first cleanly and the second only by coincidence, and that’s as far as I can take it.

Two candidates rather than a longer list, because a 1G link anywhere in the switched path is ruled out: the Mac’s end by the ifconfig output above, the NAS’s end by the fact that 700 MB/s later crossed it. What’s left is a mount bound to the wrong port, or the gateway. And I can’t tell you how fast that gateway forwards between subnets, or whether IDS/IPS was on — the common explanation in UniFi circles is the inspection stack taxing the forwarding path, and I tested none of it, so I have a path rather than a mechanism.

If route get comes back clean

If gateway: is empty and there’s no GATEWAY flag, your two machines are already talking over the switch and none of the routing story above is your problem. The candidates then are your disks, a dirty link showing up in the error counters, or a mount older than your current interface layout — the built-in-port scenario is a same-subnet one, so a clean result doesn’t exclude it.

Mostly, though, reset the expectation. On my array roughly 300 MB/s looks like the disks’ own read speed, with bursts to 700 when the cache happened to be serving. Line rate to a NAS full of spinning disks was never the number to expect, and how close you get scales with how many spindles you have.

Why the NAS was somewhere stupid

When I set the network up I had the idea that it’s a UniFi device, so all UniFi hardware should go on the management network. That doesn’t hold. A management network is for switch consoles and access point control planes; the NAS is a server that happens to be made by the same vendor as the switch, and its data plane has no more business there than the Mac’s does.

The filing decision came with a trap as well. UniFi can hand a network’s inter-VLAN routing to an L3-capable switch instead of the gateway, which is the architecturally correct fix here; Ubiquiti’s Layer 3 Routing article describes it as “setting the Router option on a new or existing Virtual Network to the UniFi Switch.” It documents two costs worth knowing before you plan around it: “Traffic Identification and features that rely on it are not supported on networks managed by an L3 switch,” and enabling it auto-creates a transit network on a VLAN ID the same page says cannot be changed.

Note the words Virtual Network. On my controller the Default network offered no way to select the switch, and that restriction isn’t documented anywhere — all 448 English articles came off the help centre’s API and got grepped as a corpus, rather than trusting a search box. Nothing says the Default network is excluded, or that it’s included. So this is something I hit, not a rule Ubiquiti states. If your storage lives there, confirm you can delegate its routing before planning a migration that assumes you can.

The confirmation, and why the bar was wrong

Before touching any architecture, the cheap test is to put both machines on one network temporarily and re-run the benchmark. A threshold got written down first: 900+ MB/s would confirm the gateway was the bottleneck.

That number wasn’t mine, and it predated anyone knowing what the array was. It came out of the Claude session early on: expect 900–1100 MB/s reads if everything is healthy. When I said the array was RAID5, the number didn’t move — it acquired a justification. Reads parallelise across the data disks, so the geometry puts the theoretical ceiling well above 10G, and the conclusion was explicit: “The bottleneck will be the network, not the array.” Writes were predicted at 300–500 MB/s.

Same network, Blackmagic Disk Speed Test, 5 GB:

  • writes 200–300 MB/s
  • reads 300–700 MB/s, fluctuating

Everything there clears the old gigabit ceiling comfortably, so putting them on one network was a real improvement. Nothing there is 900 MB/s, and the session called the diagnosis confirmed anyway on the strength of the 700 MB/s peak.

The interesting part isn’t that the bar got missed. It’s that both predictions missed the entire range: reads were predicted at 900–1100 and every reading came in under 700, writes were predicted at 300–500 and every reading came in at 300 or below. A theoretical array ceiling is not a measured one, and it will tell you the network is your limit with more confidence than it has earned.

Nor was the remaining shortfall the network. The obvious suspect for a fluctuating read range was the NAS’s NVMe cache, which at two 2 TB drives is vastly larger than a 5 GB test file — but the cache panel pointed away from it: 16% read hit rate, 7% write hit rate, 90% full. Those are appliance-wide figures accumulated over 3,207 powered-on hours rather than per-test ones, so read them as suggestive. If most reads came off the spinning array, then 300 MB/s is that array and the 700 MB/s bursts are the cache occasionally serving. The write path I can’t reconcile: a 7% write hit rate is hard to square with writes being absorbed by cache, and I don’t know whether the metric means something other than what I assume.

Where this actually ended

The proper fix — NAS onto its own storage network, inter-VLAN routing handed to the switch — is not done. I have a plan and no deployment.

What did ship is a DNS record for the NAS and a verified resolution path — two dig runs through both resolvers — and nothing had complained about it by the time I came back days later. Nothing points at that name yet, though; repointing the consumers is on the list of things I deliberately haven’t done.

So the honest headline is that the cheap fix was good enough to live with. Both machines still share a network, and that’s worth pricing rather than celebrating. A storage array and a workstation sharing one network is exactly the boundary I just finished arguing against, and mine is still eroded.

Takeaways

  1. Run route get <destination> before you suspect hardware. If gateway: is populated and flags: says GATEWAY, your two machines aren’t talking over the switch, whatever the switch cost — and returning a working adapter is the expensive wrong answer.
  2. Know what that command can’t see. It reports where a new connection would go, not where an established session already went.
  3. A theoretical array ceiling is not a measured one. Spindles times a sequential rate will tell you the network is your bottleneck. Price your confirmation threshold against what the array has actually produced.
  4. Check the network can delegate its routing before planning around it. Ubiquiti documents the feature; it doesn’t document which networks don’t offer it, and on my controller the Default network didn’t.
  5. “All the vendor’s devices belong on the management network” is not a design principle. It’s a filing habit, and it put a storage array on a control plane where it quietly cost me most of a 10G link.

Addresses and hostnames in the output above are placeholders, and the ifconfig block is trimmed: the MAC and IPv6 lines are cut. Field names, flags and column structure are as captured. Throughput figures are the ones I reported at the time, including that the original was never more precise than “gigabit speeds.” The gateway’s forwarding throughput, the IDS/IPS state and the interface error counters were never measured; the NAS’s uplink was never measured directly either, though the 700 MB/s reads put a floor on it well above gigabit. I didn’t record OS or firmware versions that night, so the UniFi UI behaviour above isn’t pinned to a version of an actively updated feature — re-check it against your own controller.