NHacker Next
  • new
  • past
  • show
  • ask
  • show
  • jobs
  • submit
Splitting the Ping (blog.benjojo.co.uk)
great_wubwub 1139 days ago [-]
Everybody is talking about clock accuracy and totally missing that devices in the middle of the network path do not care about responding quickly to pings. Middle devices are generally routers or firewalls, and their job is to route and firewall, not to respond to a packet as quickly as it comes in. Transit traffic is far more important than processing control plane packets. Devices can add several msec or more in latency by sticking ICMP echo requests and the like in a low-priority queue and getting around to responding eventually. This will dwarf any gains produced by "the best NTP server".

And no, setting QoS bits on the packet will not help.

walrus01 1139 days ago [-]
Absolutely this. One of the first thing that an ISP's NOC will tell a business customer, when they're complaining about ICMP loss or high latency to some intermediate-hop seen in a traceroute, is to test against the ICMP loss and latency/jitter to an endpoint destination.

Preferably to an endpoint destination that is some sort of server which isn't firewalling/rate limiting answering ICMP, and not a firewall or router.

You can see what looks like terrible loss and jitter to intermediate hops in a traceroute, but traffic to your your VoIP server that is 12.3ms away on the far side of those hops and several AS-to-AS adjacencies, might average 0.00% loss over multi week periods, with less than 0.3ms of jitter range.

Customers will also be advised to use other tools that can measure success/failure and RTT answer time of other services and daemons running on the endpoint they're testing against (at OSI layers 4-7), such as the various ways of measuring DNS lookup query time, or time to curl a sample file from an httpd over TLS1.2/1.3, and plot that on a multi day/week chart.

For their own protection routers and other network equipment with packets flowing through them deprioritize answering ICMP.

fragmede 1139 days ago [-]
Which is frustrating enough, but I’ve heard rumors that AWS’ (and other cloud) providers sometimes do weird things with routing, so ICMP doesn’t even take the same path as TCP traffic. Other times ICMP is just blocked somewhere along the way. I recommend TCPPing or similar for doing investigations.
notyourday 1139 days ago [-]
Just wait until you see the fun they have with edge termination where they terminate ICMP at the edge while letting TCP/UDP into the core.
bogomipz 1139 days ago [-]
>"Everybody is talking about clock accuracy and totally missing that devices in the middle of the network path do not care about responding quickly to pings. Middle devices are generally routers or firewalls, and their job is to route and firewall, not to respond to a packet as quickly as it comes in."

That's not correct. A box in the middle of the network path by definition doesn't respond at all to the ping request since it's not addressed to them. It's simply forwards the IP packet that encapsulates the ICMP echo request towards its destination. As forwarding happens in the data plane this doesn't involve the control plane at all. A router with no QoS will forward IP datagrams at the same rate whether they encapsulate TCP, UDP or ICMP.

zaphirplane 1139 days ago [-]
The destination presumably in the parent post is the network device
bogomipz 1138 days ago [-]
Please read what I quoted from the parent where they explicitly state that "middle boxes" deprioritize pings. They are actually making this distinction between middle and end device. So no, not "presumably" at all. The exact opposite.
tyingq 1139 days ago [-]
Wouldn't some ICMP messages be important to send back quickly? Like "Fragmentation Needed"?
hansel_der 1139 days ago [-]
not really. icmp itself is a datagramm protocol i.e. there is no delivery guarantee and hence it's kinda optional (in ipv4 that is, on the other hand ipv6 trades in some robustness and places a higher emphasis on working icmp)
swinglock 1139 days ago [-]
No more important than other ICMP and not important enough to prioritize it as that would risk denial of service. Answering within milliseconds is fast enough.
rwmj 1139 days ago [-]
Do they actually do this? It sounds like it would be more effort for a device to deeply inspect the packet and sort them into queues (and to what end exactly?), rather than simply forward as fast as possible based on the destination address.
sneak 1139 days ago [-]
GP is referring to when you ping the router in the path itself, not when you send pings through it.

"Forward as fast as possible" is indeed what they do. Respond to pings (to itself) that may be in those packets is further down the list.

parliament32 1138 days ago [-]
You don't need to do any DPI because ICMP is a layer 3 protocol. It's in the same layer as looking up source/destination addresses.
pfortuny 1139 days ago [-]
I don’t know but that seems a pretty simple feature to implement in hardware.
yencabulator 1136 days ago [-]
Oh, it's even worse. I've observed bottlenecked ISPs prioritizing ping, where ICMP Echo latency and packet loss are less than the same for TCP & UDP packets (and I mean packets not inside the TCP stream). I guess the squeaky wheel got the grease, and that stopped people using ping results to complain about their slow internet connection.
pxx 1139 days ago [-]
Not entirely related, but a fun and interesting tangent: There's actually no way that we know of to measure the "one-way" speed of light, as the specific synchronization that you use (and this post uses to do its calculation) assumes that the speed of light is the same in both directions. For all we know, light travels infinitely quickly in one direction and at c/2 in the return direction.

https://en.wikipedia.org/wiki/One-way_speed_of_light

recent-ish video about this: https://www.youtube.com/watch?v=pTn6Ewhb27k

lmm 1139 days ago [-]
Isn't that a distinction without a difference? You might as well say that light travels infinitely quickly in the other direction and at c/2 in the first direction - there's no measurable difference between the two cases.

There's no observable anisotropy to spacetime, so I think it makes the most sense to treat the speed of light as the same in all directions.

hansel_der 1139 days ago [-]
of course you are right, but i think op's point was that there is no way to be sure and we are left with the assumption.
bitsen 1139 days ago [-]
That’s the stupidest argument I’ve ever heard from someone attempting to be a physicist.

The gear experiment is the simplest answer. With a gear whose diameter spans the distance between the light source and receiver, and where the speed of rotation is controlled by an atomic clock, the gear could have a small hole through it. If light sent from one side to the other while the gear is spinning is too slow, it will not make it through the gear.

The size of the bike through the gear will dictate the speed, because the gear itself measures duration and distance in a single direction.

quiescant_dodo 1139 days ago [-]
I think you missed the point.

Measuring one-way speed of light requires perfectly synchronized clocks. You cannot do this over any distance, as it requires the speed of light to propagate between the clocks. So you must synchronize the clocks when they are together.

But then you must move at least one clock, which subjects the clocks to unknown time dilation. We can't even assume that the time dilation from movement X is the same as from movement -X, even to impossibly-precise measurements and controls. The _premise_ of the question about one-way speed of light is un-divorceable from time dilation. If you _assume_ that the time dilation is the exact same, you are also _assuming_ that the amount of time light takes from A-to-B is exactly half of A-to-B-to-A.

Einstein himself articulated this problem. He's certainly known for more than "attempting to be a physicist".

bitsen 1138 days ago [-]
A physicist came up with the gear solution which was stated in the video and dismissed because the mock physicist didn’t understand why the gear or wheel solved the problem.

If the gear or wheel is moving at a known rate, it can define the minimum speed of light passing through it, if the gear goes sufficiently fast and is sufficiently large with a sufficiently small hole.

Einstein was not infallible or all-knowing. His insistence on having two clocks when all that is needed is one time-accurate gear just shows he wasn’t thinking clearly. Einstein was clearly not an idiot.

However the mock physicist ignoring the very solution he presented is an idiot.

justin66 1138 days ago [-]
> Measuring one-way speed of light requires perfectly synchronized clocks.

But disproving the oversimplification provided above, For all we know, light travels infinitely quickly in one direction and at c/2 in the return direction, only requires clocks.

pxx 1138 days ago [-]
But it doesn't. The experimental procedure seems to involve a spinning disk that either allows light to travel through its entire diameter or not in an instantaneous fashion. I challenge you to try to fabricate such a device.

You'll run into an obvious problem that the disk doesn't have infinite rigidity. Otherwise you could solve the one way speed of light problem by simply poking a button at the receiver's clock with a long stick. Clearly the wave propagation through the long stick happens much slower than the light getting there...

justin66 1137 days ago [-]
Disproving light travels infinitely quickly in one direction is easier than you think and does not require the apparatus you've described. Perfect synchronization and extremely high resolution are definitely not needed to measure that one...
pxx 1137 days ago [-]
Can you describe or point to a simpler (or any) apparatus that would be capable of this result?
ramses0 1139 days ago [-]
That's actually a super interesting point and problem! One of my favorite museum visits was "the museum of time" at Greenwich (of GMT fame). Well worth the visit!

A summary of the difficulty and "technology" of time + clocks.

1) Latitude (up+down) is easy to calculate + agree upon anywhere in the globe (but what does this have to do with time, you say?). To calculate latitude, you (mumble mumble) look at where the sun is at high noon, do some math and basically figure out if you're in a "small circle" (near the poles) or "big circle" (near the equator). Effectively, latitude is "defined" as "how big around is the ring you are currently on". https://commons.wvc.edu/rdawes/ASTR217/Gnomon.pdf

2) Conversely: longitude is nigh impossible to determine, and very few people think about why that might be. (and why it has huge commerce and military implications).

Why is it difficult to determine? ...well... what is "zero"? For the equator, it's well-defined. The biggest ring around, getting smaller as you go "north" or "south" (towards the poles).

For longitude... is it "how far away from mount everest?" "how far away from new york city?" "how far away from greenwich?" (Turns out, we've settled on the last definition).

Once you've gotten farther away from visible land (can't see the tallest tree/mountain any more) you don't know how far _away_ you are from anything (east-west) ... but you can trivially determine how north/south you are (look at the sun, do some math, figure out which "ring" you're on).

...but _wait_! You _CAN_ figure out how far _AWAY_ you are from something. If you "steal" the sun at high noon in greenwich, wait 24 "true clock hours" (in greenwich), and you'd expect the sun at high noon (12:00pm on your watch) to be exactly in the same spot. Go 100 miles west, and the sun would show up a little late (compared to your 100% accurate "true clock watch"). Go 1000 miles west, and it'll be quite a bit later until the sun shows up at exactly solar noon (overhead, casts no shadow). Figure out exactly "how late is the sun?" and bam, you've calculated "how far away you are from greenwich".

Summary: North/South is "easy" (how big around is the globe). East/West is "hard", and the first, best way to measure it was "how many hours east/west are we from greenwich".

Military and Commerce implications? Accurate maps. If you go west for a month, and run into a spot of land... you know your "Y" (north/south), but you may not know your "X (east/west). If you send 10 boats with 10 inaccurate clocks going "west", they may have wildly varying calculations as to what their "west" is, depending on how accurate their _clocks_ were!

So it turns out that if you ever really do get sent back in time, and happen to be wearing a decent watch, you could become godly rich simply by being able to produce maps that are accurate in the east-west direction and not just the north-south direction (since early on, you're competing with boats carrying grandfather clocks, or burning 24 candles that each last an hour) as a mechanism to keep track of when noon is when they left, so they can calculate how far east/west they've gone.

Spoiler alert, eventually somebody figured out some other fancy math involving maybe the moon or something in order to be able to calculate a longitude, but I believe that was after or near co-incidental with more accurate spring-wound clocks which reduced the importance of being able to do it simply via calculation.

Nowadays we take GPS for granted, but it's surprising how important of a problem it's actually solving.

jlgaddis 1139 days ago [-]
On a somewhat related note, most people aren't aware that you can easily figure out direction (i.e., "which way is North?") using just your wristwatch -- an analog one, not your smart watch!

I learned that one as a kid in Boy Scouts and it somehow stuck in my mind -- and there were a few times throughmy life that it came in handy.

Of course, it's less useful now that we all have a smartphone with a compass GPS in our pockets.

(And if you're unlucky enough to not be wearing your wristwatch when you get lost, you can find your way using nothing more than a stick in the ground -- but this method will take longer as you'll have to wait 20 or 30 minutes.)

(And if it's night -- meaning you can't use either of the two methods above, you'll have to rely on the North Star.)

chrisshroba 1138 days ago [-]
Really leaving us hanging! How do I find out which way is North using my wristwatch or a stick?
reasonabl_human 1138 days ago [-]
What is the wrist watch trick? Keeping time and looking at the sun / shadows?
jtbayly 1138 days ago [-]
That inability to figure out longitude also led to thousands of people dying when ship captains had to guess whether they were west or east of the island they needed to hit for water and food resupplies.
Ono-Sendai 1139 days ago [-]
It's basically impossible to measure the one-way latency, without external information in the form of clock synchronisation done externally. See http://twistedoakstudios.com/blog/Post2353_when-one-way-late...

This fact is also the basis for special relativity (different observers may choose different simultaneity conventions - for example Einstein synchronisation)

datastoat 1139 days ago [-]
To be precise, it's impossible to measure one-way latency between a pair of nodes without external information. But if you have a mesh of nodes, the story is different [1]. If you have N nodes and hence N unknown clock offsets, and if you have ping data from N(N-1) pairs, you can do better.

[1] https://www.usenix.org/conference/nsdi18/presentation/geng

Ono-Sendai 1139 days ago [-]
Well I think we need to distinguish between practical latency measurements on earth, where it's mostly a matter of rejecting dodgy data / outliers etc.., and detecting systematic latency bias that is theoretically impossible to detect. If you could get around this problem just by using a network of N nodes, then you could do stuff like measuring an absolute reference frame, thereby violating special relativity.
bentcorner 1139 days ago [-]
You should be able to tell when one side of the link degrades though, correct? You'd have known time deltas for the link each way (whether its accurate doesn't really matter) and when the round trip time changes you should be able to tell which side that occurred on. (I'm interested in this problem because my cable internet reliably (!) degrades during the workday)
slaymaker1907 1139 days ago [-]
The problem is physics can't even distinguish whether light travels instantaneously in direction x, but at c/2 going in the opposite direction. There is no known way to synchronize clocks or come up with a clever apparatus to prove that light moves at the same speed in every direction.

You need to make some strong assumptions in order to try and measure asymmetrical latency on a network. I believe the method you are proposing is to measure RTT when RTT differs depending on the starting node. The difficult problem is determining how much of your RTT time comes from node A to node B as opposed to node B to node A. I believe you need to at least have some sort of oracle on both sides which can synchronize time without using the network path you are trying to measure (I think you can do this if you have node C which both A and B have a fast connection to relative to each other, this implies that the delay between A and B is significantly longer than the speed of light delay).

nullserver 1139 days ago [-]
https://web.mit.edu/jemorris/humor/500-miles

Email can’t go father then 500 miles

datastoat 1139 days ago [-]
As the article explains, latency and clock sync go hand in hand. Here's a blog post [1] that goes further into clock sync, contrasting NTP and hardware-based systems. The company behind the blog post says that their solution is available as a managed service on Azure and GCP, but I've never looked out for it.

[1] https://www.ticktocknetworks.com/tick-tock-the-clock-runs-wi...

jlgaddis 1139 days ago [-]
If you've got clock synchronization between two hosts, there's OWAMP, a.k.a. RFC4656 [0].

The Minimum-Pairs Protocol [1] eliminates the need for clock synchronization but requires (at least) 3 hosts under your control, plus the "uncooperative" fourth node:

> The minimum-pairs (or MP) is an active measurement protocol to estimate in real-time the smaller of the forward and reverse one-way network delays (OWDs).[1] It is designed to work in hostile environments, where a set of three network nodes can estimate an upper-bound OWDs between themselves and a fourth untrusted node. All four nodes must cooperate, though honest cooperation from the fourth node is not required. The objective is to conduct such estimates without involving the untrusted nodes in clock synchronization, and in a manner more accurate than simply half the Round-Trip Time (RTT).

--

[0]: https://tools.ietf.org/html/rfc4656

[1]: https://en.wikipedia.org/wiki/Minimum-Pairs_Protocol

jedimastert 1139 days ago [-]
An excellent video about why it's impossible to measure the speed of light in one direction

https://www.youtube.com/watch?v=pTn6Ewhb27k

1139 days ago [-]
IgorPartola 1139 days ago [-]
I thought NTP specifically did account for transit latency. Not sure why I made that assumption, but if it’s not true, how can I ever trust my clock to be correct?
detaro 1139 days ago [-]
As the article says, it does account for it. It just can't account for unknown constant asymmetric latency.

> how can I ever trust my clock to be correct?

what does it mean for your clock to be "correct"? If you need your time to be precise to more than a few 100ms you probably shouldn't be getting it from random NTP servers over unknown connections, but for most people that's an acceptable error.

swinglock 1139 days ago [-]
This protocol and software exists, it's called OWAMP.
chlbny 1139 days ago [-]
OWAMP is even a standardized protocol (RFC 4656).

For the record, there is another tool that can perform such kind of one-way measurement called https://github.com/heistp/irtt .

statstutor 1139 days ago [-]
The article says "Now that we have a synced clock", but surely the clock synchronisation would also be unable to resolve any asymmetry in the ping and the synchronisation would inevitably be impacted by this. (Or, does NTP have a solution to this?)

[Edited to add: from https://en.wikipedia.org/wiki/Network_Time_Protocol: "Asymmetric routes... can cause errors of 100 ms or more."]

jtsiskin 1139 days ago [-]
Yes, NTP clock sync assumes symmetric delays, so you have the exact same problem.

However they use the closest NTP time server, so assuming the NTP server owners correctly are able to sync their own servers, the offset is probably close.

Unless the asymmetric delay is on the path which both the ping and the NTP synchronization take

toast0 1139 days ago [-]
> Unless the asymmetric delay is on the path which both the ping and the NTP synchronization take

Assuming the NTP server is not on the LAN using a GPS (or other radio) reference, a large part of the asymmetry is on the poster's VDSL last mile hop. At least on my bonded VDSL2 line, it seems there's significant asymmetry; I found with regular ntpd I needed to delay incoming packets by around 18 ms to get a good match between a GPS receiver and network servers; my round trip time to the PPPoE concentrator is about 21 ms, and I was able to find a few NTP servers close enough that round trip to them was about 22 ms; not much room for additional asymmetry there.

benjojo12 1139 days ago [-]
The point was to say, if you are in a DC environment and have ~1ms access to a Stratum 1 NTP server, then you can use that, if you are on DSL/DOCSIS etc, then it's likely required to use GNSS/PPS sources
dec0dedab0de 1139 days ago [-]
This reminded me of the buffer bloat[0] problem esr was talking about a few years ago.

I thought I read a paper where he was getting hardware made for the purpose of using gps for time so it could be accurate enough to measure latency. I just googled and I realized it wasn't a paper, it was a talk I went to in 2012 that I forgot about[1]

I wonder if anything came of it, does anyone know?

[0] https://hn.algolia.com/?q=bufferbloat (it made the front page more than once)

[1] https://youtu.be/1b17ggwkR60

drothlis 1139 days ago [-]
This "not quite 5 minute guide to making an NTP server" is from 2014: https://ava.upuaut.net/?p=726

So GPS boards for Raspberry Pi were cheap off-the-shelf hardware by 2014, at least.

gerdesj 1139 days ago [-]
"However there is a common assumption on this latency number. Is that you can divide it in half to get the time it takes to send data in one direction"

So you ssh the other end and ping in the other direction. I tend to use mtr instead.

Nowadays latency to internet services are within the sort of latencies we used to require on site. OK I do understand that not all internet connections are equal. I live in a fairly rural part of the UK - a small town of roughly 25,000 odd people. The UK is a fairly small country and fairly densely populated and quite rich, so in general: internets are fairly reasonable in comparison to the rest of the world. Not world beating but overall pretty decent.

If I ping say www.google.com (yes I know it's a funky address with random "distance") from home on my FTTC connection I see something like 15 to 20ms returns. I can ssh the office and use a 1GB link and get 8ms returns. My office PC is on the end of three 1GBs-1 hops in the building itself and it's a crappy old ex-customer hand me down (I run Arch Linux, I don't need whatever W10 requires). Both of those links are via the same ISP. My home PC test is via wifi!

I'm not going to bother going too much further with this now but if I was bothered (and I will be one day!), I would start my pinging n stuff at my internet facing router.

It wasn't that long ago that we (my company, about 10 years ago) accepted a condition of contract that required a site latency of 30ms end to end (ignoring hosts - the latency was directly measured without normal IT involvement.) 30ms!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Rendered at 00:43:50 GMT+0000 (Coordinated Universal Time) with Vercel.