Still using ping to test connectivity? Try mtr!

🛠 Tools and Programming
1

MTR stands for My TraceRoute, originally derived from Matt’s TraceRoute. mtr is a network diagnostic tool that combines the functions of the ping and traceroute commands.

The ping command focuses on checking the latency to the target node.

ping 8.8.8.8 
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 
64 bytes from 8.8.8.8: icmp_seq=2 ttl=106 time=245 ms 
64 bytes from 8.8.8.8: icmp_seq=4 ttl=106 time=245 ms 
... ...

The traceroute command can view information about every node (or hop) the packet passes through.

traceroute -n 8.8.8.8 
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets 
1 192.168.1.1 1.229 ms 1.143 ms 1.055 ms 2 117.30.114.1 133.626 ms 133.606 ms 134.264 ms 
... ... 
12 108.170.237.45 252.366 ms 209.85.252.245 282.760 ms 209.85.240.31 282.729 ms 
13 8.8.8.8 241.518 ms 241.971 ms 250.866 ms

For each node the packet passes through from the source to the destination, mtr records it. Then it sends ICMP packets to these nodes and obtains network information of these nodes (such as latency, packet loss rate, etc.) through the ICMP response packets. In most cases, the packets mtr uses to detect network information are ICMP packets, but they can also be TCP or UDP packets.

Original Text

This article is converted by SimpRead, original URL www.cnblogs.com

References:

MTR (software) - Wikipedia

MTR Official Website

mtr man manual

Introduction

MTR stands for My TraceRoute, originally derived from Matt’s TraceRoute. mtr is a network diagnostic tool that combines the functions of the ping and traceroute commands.

The ping command focuses on checking the latency to the target node.

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=2 ttl=106 time=245 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=106 time=245 ms
... ...

The traceroute command can view information about every node (or hop) the packet passes through.

[](javascript:void(0); “Copy code”)

# traceroute -n 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  192.168.1.1  1.229 ms  1.143 ms  1.055 ms
 2  117.30.114.1  133.626 ms  133.606 ms  134.264 ms
 ... ...
12  108.170.237.45  252.366 ms 209.85.252.245  282.760 ms 209.85.240.31  282.729 ms
13  8.8.8.8  241.518 ms  241.971 ms  250.866 ms

[](javascript:void(0); “Copy code”)

For each node the packet passes through from the source to the destination, mtr records it. Then it sends ICMP packets to these nodes and obtains their network information (such as latency, packet loss rate, etc.) through ICMP response packets. In most cases, mtr uses ICMP packets to detect network information, but TCP or UDP packets can also be used.

We can type the following command to try.

Note: When I use Vmware Workstation Pro 15.5 to build a virtual machine, if the virtual machine’s network is in NAT mode, then mtr tracing Baidu returns very few results, only one or two lines. Switching to bridged mode works fine. If anyone knows how to solve this problem, please kindly advise!

# mtr -n www.baidu.com

The result is as follows. The fullscreen will be replaced with mtr’s output. By default, mtr sends ICMP packets every second to probe the network condition, so the interface refreshes once every second.

In this interface, we can enter various commands to interact with the mtr results, such as “p” to pause, space to continue, etc. This user-interactive mode/interface is generally a text user interface (TUI) based on curses (a terminal control library).

Next, we will discuss the command options, interactive commands with the command results, and how to interpret mtr results.

Command Options

Operating system and software version:

# lsb_release -d
Description:    CentOS Linux release 7.5.1804 (Core) 
# mtr --version
mtr 0.85

The man manual syntax is somewhat complex because it lists both short and long options.

mtr   [-BfhvrctglxspQemniuTP46]   [--help]  [--version]  [--report]  [--report-wide]  [--report-cycles COUNT]  [--curses]
       [--split] [--raw] [ --xml] [--mpls] [--no-dns] [--show-ips] [--gtk] [--address IP.ADD.RE.SS] [--interval SECONDS] [--max-
       ttl NUM]  [--first-ttl NUM]  [--bitpattern NUM]  [--tos NUM]  [--psize BYTES  |  -s  BYTES] [--tcp] [--udp] [--port PORT]
       [--timeout SECONDS] HOSTNAME [PACKETSIZE]

We can roughly simplify it to this, then understand each option one by one.

mtr [OPTIONS] HOSTNAME

-h, --help: Shows a brief description of the command syntax.

-v, --version: Shows the version. This is important because different versions of mtr may output different results.

-r, --report: Use report mode. As demonstrated above, mtr by default enters the TUI interface. If we don’t want mtr to enter TUI and just want it to output results like other commands, we can use report mode.

[](javascript:void(0); “Copy code”)

# mtr -r 8.8.8.8
Start: Fri Mar  5 10:56:20 2021
HOST: c7-server                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- gateway                    0.0%    10    1.4   1.8   1.1   3.4   0.7
  2.|-- 1.114.30.117.broad.xm.fj.  0.0%    10   65.1  24.2   1.6 115.9  38.4
  3.|-- 218.85.117.189             0.0%    10    2.5   2.9   2.1   5.1   0.7
  4.|-- 61.154.238.65              0.0%    10    3.7   5.1   3.5   7.0   1.2
  5.|-- 202.97.82.221              0.0%    10   25.5  22.0  18.7  26.1   2.9
  6.|-- 202.97.94.237             20.0%    10   33.6  33.9  32.1  36.2   1.1
  7.|-- 202.97.12.186             70.0%    10   19.3  19.6  19.3  20.1   0.0
  8.|-- 202.97.50.38               0.0%    10  211.0 218.8 209.9 233.7   8.6
  9.|-- 118.85.205.234             0.0%    10  217.1 213.9 205.1 222.2   6.2
 10.|-- 72.14.217.174              0.0%    10  243.0 245.9 234.9 262.2   7.1
 11.|-- 108.170.241.193           20.0%    10  255.2 256.3 255.2 259.4   1.3
 12.|-- 142.250.224.131            0.0%    10  256.4 255.8 254.3 256.5   0.5
 13.|-- dns.google                10.0%    10  244.6 244.7 243.8 245.5   0.4

[](javascript:void(0); “Copy code”)

Here is a brief analysis of each field in this report:

  1. From 1 to 13, an incremental sequence number, indicating which hop the packet passed through. From the last hop number, we can also know how many hops the packet passed in total.
  2. The IP address of the routing node passed, resolved to hostname if reverse resolution is supported.
  3. Loss%: Packet loss rate.
  4. Snt: Number of packets sent.
  5. Last: The latency of the most recent packet. All latency-related time units displayed by mtr are in milliseconds.
  6. Avg: Average latency.
  7. Best: Minimum latency.
  8. Wrst: Maximum latency.
  9. Stdev: Standard deviation of latency; the smaller this value, the more stable the latency.

-c, --report-cycles COUNT: Report mode by default sends only 10 packets. We can specify the number of packets to send with -c. The field Snt shows the number of sent packets.

# mtr -c 5 -r 8.8.8.8

-w, --report-wide: Wide report mode. In the default report mode, if the hostname is too long, it will be truncated.

1.114.30.117.broad.xm.fj.    # Without wide report mode
mtr -w 8.8.8.8
1.114.30.117.broad.xm.fj.dynamic.163data.com.cn    # With wide report mode

-s, --psize BYTES: Specify packet size in bytes. You can also specify PACKETSIZE at the end of the command line.

mtr -s 100 8.8.8.8
mtr 8.8.8.8 100    # This specification method produces no result, unknown reason.

-t, --curses: Force mtr to use the curses library TUI interface, which is the default option.

-e, --mpls: Should make mtr show ICMP extended MPLS information. MPLS is a multi-protocol encapsulation technology that speeds up routing by avoiding routing table lookups. This is an advanced feature that I am not very familiar with.

-n, --no-dns: Do not resolve hostnames. IP addresses won’t be reverse-resolved to hostnames, which speeds up the return of mtr results.

mtr -n -r 8.8.8.8
gateway <==> 192.168.1.1
1.114.30.117.broad.xm.fj. <==> 117.30.114.1
dns.google <==> 8.8.8.8

-b, --show-ips: Show both hostname and IP address. Can be used in split mode and report mode. If the hostname gets truncated in report mode, wide report mode is used.

-g, --gtk: Use GTK-based GUI, generally not used.

-p, --split: Split mode, suitable for split-user interface. I don’t quite understand what it means. Output is similar to fields separated by spaces.

# mtr -p -c 1 8.8.8.8

-l, --raw: Output raw mode, which can be parsed by other programs. Generally not used.

-x, --xml: Output in XML format.

-a, --address IP.ADD.RE.SS: Specify which IP address the packet uses to exit when mtr probes. However, this option cannot be used for DNS requests; otherwise, results may be unexpected.

-i, --interval SECONDS: Specify the interval between sending ICMP ECHO request packets in seconds, default is 1 second. This can be any positive number, not necessarily an integer.

mtr -i 1 8.8.8.8
mtr -i 0.5 8.8.8.8

-m, --max-ttl NUM: Specify the TTL value of the packets we send. mtr uses this value to send packets to each node in the routing path. For example, in our example, there are 13 nodes total, so if setting TTL, it must be at least 13, or the packet won’t reach 8.8.8.8.

mtr -m 1 -r -c 1 8.8.8.8    # Result with 1 is somewhat odd
mtr -m 2 -r -c 1 8.8.8.8
mtr -m 5 -r -c 1 8.8.8.8
mtr -m 12 -r -c 1 8.8.8.8

-f, --first-ttl NUM: Specify from which node to start probing, default is from the first node (gateway).

mtr -f 5 -r -c 1 8.8.8.8

-B, --bitpattern NUM: Specify the bit pattern in the payload, with a value between 0 and 255. My understanding of bit pattern here is multi-bit binary data, which could be 4 bits, 8 bits, 16 bits, etc., collectively representing something; for example, 4 bytes represent an integer int. So the bit pattern would be 4*8=32.

-Q, --tos NUM: Specify the value of the type of service field in the IP header, ranging from 0 to 255. The type of service here refers to the type of service, now called DSCP, which is a field in the IP header. DSCP is generally used for some real-time data streams, such as IP telephony.

-u, --udp: Use UDP datagrams instead of ICMP ECHO.

-T, --tcp: Use TCP SYN packets instead of ICMP ECHO. When using this option, the options to modify packet size -s, --psize, or PACKETSIZE will be ignored because SYN packets do not contain data.

-P, --port PORT: Specify the target port when using TCP tracing.

–timeout SECONDS: Timeout for keeping the TCP socket open (disconnect on timeout). This only affects the last hop. If this value is large and the interval (-i) is short, a large number of file descriptors will be used.

-4 and -6: Display only ipv4 or ipv6, options rarely used because ipv4 is still widely used now.

-o, --order fields order: Specify the fields information to output. This option is an important one.

  • L: lose ratio, packet loss rate.
  • D: dropped packets.
  • R: received packets.
  • S: sent packets.
  • N: newest RTT, the latest latency, in milliseconds. RTT means Round-Trip Time, representing the time for a packet to go and return.
  • B: best/min RTT, the best/minimum latency.
  • A: average RTT, average latency.
  • W: worst/max RTT, worst/maximum latency.
  • V: standard deviation, the standard deviation of latency. Indicates how much each delay differs from the average delay. The smaller the value, the more stable the network.
  • G: getmetric mean, geometric mean. Different from arithmetic mean, understanding it is a challenge.
  • J: current jitter.
  • M: mean/average jitter.
  • X: worst jitter.
  • I: interarrival jitter.

Report Analysis

References:

Diagnosing Network Issues with MTR | Linode

How to Use MTR to Diagnose Network Problems | HelloDog

Unless otherwise specified, the mtr output in the Results Analysis comes from Linode.

[](javascript:void(0); “Copy code”)

% mtr --no-dns --report google.com
HOST:  deleuze                       Loss%   Snt   Last  Avg  Best  Wrst   StDev
    1. 192.168.1.1                   0.0%    10    2.2   2.2   2.0   2.7   0.2
    2. 68.85.118.13                  0.0%    10    8.6  11.0   8.4  17.8   3.0
    3. 68.86.210.126                 0.0%    10    9.1  12.1   8.5  24.3   5.2
    4. 68.86.208.22                  0.0%    10   12.2  15.1  11.7  23.4   4.4
    5. 68.85.192.86                  0.0%    10   17.2  14.8  13.2  17.2   1.3
    6. 68.86.90.25                   0.0%    10   14.2  16.4  14.2  20.3   1.9
    7. 68.86.86.194                  0.0%    10   17.6  16.8  15.5  18.1   0.9
    8. 75.149.230.194                0.0%    10   15.0  20.1  15.0  33.8   5.6
    9. 72.14.238.232                 0.0%    10   15.6  18.7  14.1  32.8   5.9
   10. 209.85.241.148                0.0%    10   16.3  16.9  14.7  21.2   2.2
   11. 66.249.91.104                 0.0%    10   22.2  18.6  14.2  36.0   6.5

[](javascript:void(0); “Copy code”)

Each line starting with a number represents a routing node the packet went through, also called a hop. The main data we care about are latency and packet loss. Latency is mainly judged by the average delay Avg. The standard deviation indicates whether the average delay information is closer to the true situation; the larger the standard deviation, the lower the reliability of the average delay.

Generally, we can divide all routing nodes into three segments:

  • The first 2-3 nodes are usually the local ISP segment. Assuming we run mtr on the user’s PC, this is the customer’s local ISP segment, e.g., home/community broadband/fiber. If abnormalities appear in this segment, we should contact local ISPs like China Telecom or China Mobile.
  • The last 2-3 nodes are usually the target ISP segment, typically providers of services, such as the Taobao network where we shop, or the game server network operated by the game provider. If abnormalities appear in this segment, we should contact Taobao or the game customer service technical staff.
  • The middle nodes may be many, usually involving some backbone networks spanning cities, provinces, countries or continents of the wide area network. If there is a problem in the middle nodes, usually neither the local ISP nor the target ISP can do much.

Packet Loss

Generally, if we see a sudden high loss percentage at a certain node in the mtr output, we might think that node has an issue. However, most routing nodes reduce or even do not respond to ICMP echo request packets because such packets are usually used for network probing rather than real network traffic. Therefore, packet loss alone does not represent actual packet loss; mtr’s ICMP packets might be 100% dropped, but real traffic packets may not be lost. To judge whether it’s ICMP traffic limiting or real packet loss, we can see if subsequent nodes after the loss node also experience loss. If subsequent nodes show no loss as below, the loss is likely due to ICMP traffic limiting.

[](javascript:void(0); “Copy code”)

HOST: example                       Loss%   Snt   Last  Avg  Best  Wrst   StDev
   1. 63.247.74.43                  0.0%    10    0.3   0.6   0.3   1.2   0.3
   2. 63.247.64.157                50.0%    10    0.4   1.0   0.4   6.1   1.8
   3. 209.51.130.213                0.0%    10    0.8   2.7   0.8  19.0   5.7
   4. aix.pr1.atl.google.com        0.0%    10    6.7   6.8   6.7   6.9   0.1
   5. 72.14.233.56                  0.0%    10    7.2   8.3   7.1  16.4   2.9
   6. 209.85.254.247                0.0%    10   39.1  39.4  39.1  39.7   0.2
   7. 64.233.174.46                 0.0%    10   39.6  40.4  39.4  46.9   2.3
   8. gw-in-f147.1e100.net          0.0%    10   39.6  40.5  39.5  46.7   2.2

[](javascript:void(0); “Copy code”)

If multiple subsequent nodes also have loss, it may indicate actual packet loss. Remember, ICMP traffic limiting and packet loss can occur simultaneously!

In this example, we consider the minimum loss rate in the subsequent nodes to be the actual packet loss rate, which is 40%.

[](javascript:void(0); “Copy code”)

HOST: localhost                      Loss%   Snt   Last  Avg  Best  Wrst   StDev
   1. 63.247.74.43                   0.0%    10    0.3   0.6   0.3   1.2   0.3
   2. 63.247.64.157                  0.0%    10    0.4   1.0   0.4   6.1   1.8
   3. 209.51.130.213                60.0%    10    0.8   2.7   0.8  19.0   5.7
   4. aix.pr1.atl.google.com        60.0%    10    6.7   6.8   6.7   6.9   0.1
   5. 72.14.233.56                  50.0%    10    7.2   8.3   7.1  16.4   2.9
   6. 209.85.254.247                40.0%    10   39.1  39.4  39.1  39.7   0.2
   7. 64.233.174.46                 40.0%    10   39.6  40.4  39.4  46.9   2.3
   8. gw-in-f147.1e100.net          40.0%    10   39.6  40.5  39.5  46.7   2.2

[](javascript:void(0); “Copy code”)

Node 3 shows packet loss rising from 0% to 60%, and subsequent nodes do not have zero loss, so it can be judged that real packet loss begins at this node. Nodes 3 and 4 have 60% loss, with subsequent nodes gradually decreasing to 40% at the last node. Generally, the lowest loss rate at the end is selected as the real packet loss rate. According to the results, the real packet loss rate should be around 40%, with middle nodes possibly having about 10%~20% loss due to ICMP traffic limiting. This 10% to 20% is the difference between the node loss and the real loss.

Sometimes packets are sent out fine, but issues occur during the return path. It is generally recommended to perform bi-directional mtr tests for combined network diagnosis.

Since network protocols were originally designed with some elasticity for network degradations, and routing may fluctuate due to various issues, do not obsess over every packet loss. About 10% packet loss is not a main concern and may be compensated at the application layer.

Latency

In general, latency increases with distance and number of nodes. Within the same network/region, latency usually increases steadily and linearly. Latency is heavily dependent on the line quality during testing; thus, it’s best to have baseline latency records under normal conditions for comparison. Normal condition here means the software/application can function fully under the current latency.

Line quality can also be affected by the physical layer; for example, dial-up lines have higher latency than the widely used twisted pair or fiber optic cables.

[](javascript:void(0); “Copy code”)

root@localhost:~# mtr --report www.google.com
HOST: localhost                      Loss%   Snt   Last   Avg  Best  Wrst  StDev
    1. 63.247.74.43                  0.0%    10    0.3   0.6   0.3   1.2   0.3
    2. 63.247.64.157                 0.0%    10    0.4   1.0   0.4   6.1   1.8
    3. 209.51.130.213                0.0%    10    0.8   2.7   0.8  19.0   5.7
    4. aix.pr1.atl.google.com        0.0%    10  388.0 360.4 342.1 396.7   0.2
    5. 72.14.233.56                  0.0%    10  390.6 360.4 342.1 396.7   0.2
    6. 209.85.254.247                0.0%    10  391.6 360.4 342.1 396.7   0.4
    7. 64.233.174.46                 0.0%    10  391.8 360.4 342.1 396.7   2.1
    8. gw-in-f147.1e100.net          0.0%    10  392.0 360.4 342.1 396.7   1.2

[](javascript:void(0); “Copy code”)

In this example, latency dramatically increases at node 4 and remains high to the last node. We can judge there is a real latency issue. Common causes might be saturated peering sessions, misconfigured routers, or congested links, but it’s basically impossible for us to find the exact cause.

However, sometimes latency may not be a problem. For example, in the above, despite high latency, packets still reach the destination node. Latency can also be affected by return routing, and the return path may be completely different from the outgoing path.

Translator’s note: Here is a question — if bi-directional mtr tests have different paths, how valuable is their reference?

From the above example, since latency spikes at node 4 and subsequent nodes’ latency does not fluctuate significantly, we can consider the cause of the latency is at node 4.

In addition to packet loss, ICMP traffic limiting can also manifest as latency issues.

[](javascript:void(0); “Copy code”)

root@localhost:~# mtr --report www.google.com
HOST:  localhost                     Loss%   Snt   Last  Avg  Best  Wrst   StDev
    1. 63.247.74.43                  0.0%    10    0.3   0.6   0.3   1.2   0.3
    2. 63.247.64.157                 0.0%    10    0.4   1.0   0.4   6.1   1.8
    3. 209.51.130.213                0.0%    10    0.8   2.7   0.8  19.0   5.7
    4. aix.pr1.atl.google.com        0.0%    10    6.7   6.8   6.7   6.9   0.1
    5. 72.14.233.56                  0.0%    10  254.2 250.3 230.1 263.4   2.9
    6. 209.85.254.247                0.0%    10   39.1  39.4  39.1  39.7   0.2
    7. 64.233.174.46                 0.0%    10   39.6  40.4  39.4  46.9   2.3
    8. gw-in-f147.1e100.net          0.0%    10   39.6  40.5  39.5  46.7   2.2

[](javascript:void(0); “Copy Code”)

At first glance, we might think the issue is with node 5, as the latency suddenly spikes from below 10ms to around 250ms. At this point, we should apply a similar approach as we do for packet loss and examine the latency at subsequent nodes. The latency at later nodes remains around 40ms, so the actual latency is probably only about 40ms. The 250ms at node 5 is likely due to traffic throttling on ICMP packets at that node. If the latency at subsequent nodes stays at 250ms, then the latency is truly around 250ms; but if the latency at subsequent nodes drops to 0ms, then the 250ms latency at node 5 is very likely entirely caused by ICMP traffic throttling.

Common Reports

Target Host Network Misconfiguration

[](javascript:void(0); “Copy Code”)

root@localhost:~# mtr --report www.google.com
HOST:  localhost                     Loss%   Snt   Last  Avg  Best  Wrst  StDev
    1. 63.247.74.43                  0.0%    10    0.3   0.6   0.3   1.2   0.3
    2. 63.247.64.157                 0.0%    10    0.4   1.0   0.4   6.1   1.8
    3. 209.51.130.213                0.0%    10    0.8   2.7   0.8  19.0   5.7
    4. aix.pr1.atl.google.com        0.0%    10    6.7   6.8   6.7   6.9   0.1
    5. 72.14.233.56                  0.0%    10    7.2   8.3   7.1  16.4   2.9
    6. 209.85.254.247                0.0%    10   39.1  39.4  39.1  39.7   0.2
    7. 64.233.174.46                 0.0%    10   39.6  40.4  39.4  46.9   2.3
    8. gw-in-f147.1e100.net         100.0    10    0.0   0.0   0.0   0.0   0.0

[](javascript:void(0); “Copy Code”)

At first glance at this report, we mainly focus on the 100% packet loss at the last node. The last node corresponds to the target network ISP segment and is the final node. Since it is the last node, we cannot determine the packet loss situation for subsequent nodes. ICMP traffic throttling usually happens at intermediate nodes and generally must be combined with the status of subsequent nodes to judge whether it is traffic throttling.

Therefore, in such cases, it is generally unlikely to be ICMP traffic throttling at intermediate routing nodes. A more likely scenario is a network configuration problem on the last node’s server (possibly). If real packets also cannot reach it, then the firewall could be misconfigured. Of course, it is also possible that the server is directly configured to refuse ICMP packet responses.

Residential or Business Routers

[](javascript:void(0); “Copy Code”)

% mtr --no-dns --report google.com
HOST:  deleuze                       Loss%   Snt   Last  Avg  Best  Wrst  StDev
    1. 192.168.1.1                   0.0%    10    2.2   2.2   2.0   2.7   0.2
    2. ???                          100.0    10    8.6  11.0   8.4  17.8   3.0
    3. 68.86.210.126                 0.0%    10    9.1  12.1   8.5  24.3   5.2
    4. 68.86.208.22                  0.0%    10   12.2  15.1  11.7  23.4   4.4
    5. 68.85.192.86                  0.0%    10   17.2  14.8  13.2  17.2   1.3
    6. 68.86.90.25                   0.0%    10   14.2  16.4  14.2  20.3   1.9
    7. 68.86.86.194                  0.0%    10   17.6  16.8  15.5  18.1   0.9
    8. 75.149.230.194                0.0%    10   15.0  20.1  15.0  33.8   5.6
    9. 72.14.238.232                 0.0%    10   15.6  18.7  14.1  32.8   5.9
   10. 209.85.241.148                0.0%    10   16.3  16.9  14.7  21.2   2.2
   11. 66.249.91.104                 0.0%    10   22.2  18.6  14.2  36.0   6.5

[](javascript:void(0); “Copy Code”)

Phenomenon: Node 2 (and the initial nodes) shows 100% packet loss and no IP address. This may be a local ISP configuration issue; the node marked with question marks may be a local residential gateway. Since there is no packet loss at subsequent nodes after the loss node, this packet loss node is probably only refusing to respond to ICMP packets. Also, I believe the reason why the node IP is not shown may also be because the node refuses to respond to certain ICMP information.

ISP Router Misconfiguration

If the target host is unreachable, and in the mtr report all nodes after a certain node are shown as ???, then it is possible that the last valid node or a router beyond it has a configuration issue causing this.

[](javascript:void(0); “Copy Code”)

root@localhost:~# mtr --report www.google.com
HOST:  localhost                     Loss%   Snt   Last  Avg  Best  Wrst   StDev
    1. 63.247.74.43                  0.0%    10    0.3   0.6   0.3   1.2   0.3
    2. 63.247.64.157                 0.0%    10    0.4   1.0   0.4   6.1   1.8
    3. 209.51.130.213                0.0%    10    0.8   2.7   0.8  19.0   5.7
    4. aix.pr1.atl.google.com        0.0%    10    6.7   6.8   6.7   6.9   0.1
    5. ???                           0.0%    10    0.0   0.0   0.0   0.0   0.0
    6. ???                           0.0%    10    0.0   0.0   0.0   0.0   0.0
    7. ???                           0.0%    10    0.0   0.0   0.0   0.0   0.0
    8. ???                           0.0%    10    0.0   0.0   0.0   0.0   0.0
    9. ???                           0.0%    10    0.0   0.0   0.0   0.0   0.0
   10. ???                           0.0%    10    0.0   0.0   0.0   0.0   0.0

[](javascript:void(0); “Copy Code”)

The ??? here feels different from the previous example. The ??? in the residential gateway example was only for a single node and involved packet loss. Here, the situation is continuous ??? from a certain node to the end, with both packet loss and latency results being zero. It should be that at node 4 the trace stops (mtr does not know what the subsequent nodes are), rather than subsequent nodes refusing to respond to mtr packets.

Some apparent cases can be observed, such as routing loops caused by misconfigured routers.

[](javascript:void(0); “Copy Code”)

root@localhost:~# mtr --report www.google.com
HOST:  localhost                     Loss%   Snt   Last  Avg   Best  Wrst  StDev
    1. 63.247.74.43                  0.0%    10    0.3   0.6   0.3   1.2   0.3
    2. 63.247.64.157                 0.0%    10    0.4   1.0   0.4   6.1   1.8
    3. 209.51.130.213                0.0%    10    0.8   2.7   0.8  19.0   5.7
    4. aix.pr1.atl.google.com        0.0%    10    6.7   6.8   6.7   6.9   0.1
    5. 12.34.56.79                   0.0%    10    0.0   0.0   0.0   0.0   0.0
    6. 12.34.56.78                   0.0%    10    0.0   0.0   0.0   0.0   0.0
    7. 12.34.56.79                   0.0%    10    0.0   0.0   0.0   0.0   0.0
    8. 12.34.56.78                   0.0%    10    0.0   0.0   0.0   0.0   0.0
    9. 12.34.56.79                   0.0%    10    0.0   0.0   0.0   0.0   0.0
   10. 12.34.56.78                   0.0%    10    0.0   0.0   0.0   0.0   0.0
   11. 12.34.56.79                   0.0%    10    0.0   0.0   0.0   0.0   0.0
   12. ???                           0.0%    10    0.0   0.0   0.0   0.0   0.0
   13. ???                           0.0%    10    0.0   0.0   0.0   0.0   0.0
   14. ???                           0.0%    10    0.0   0.0   0.0   0.0   0.0

[](javascript:void(0); “Copy Code”)

Routing configuration errors may appear at node 4 or node 5. The solution is to try to identify the ISP responsible for the erroneous node and then contact them for resolution.

ICMP Rate Limiting

This is what I refer to as traffic throttling.

[](javascript:void(0); “Copy Code”)

root@localhost:~# mtr --report www.google.com
HOST:  localhost                     Loss%   Snt   Last  Avg  Best  Wrst   StDev
    1. 63.247.74.43                  0.0%    10    0.3   0.6   0.3   1.2   0.3
    2. 63.247.64.157                 0.0%    10    0.4   1.0   0.4   6.1   1.8
    3. 209.51.130.213                0.0%    10    0.8   2.7   0.8  19.0   5.7
    4. aix.pr1.atl.google.com        0.0%    10    6.7   6.8   6.7   6.9   0.1
    5. 72.14.233.56                 60.0%    10   27.2  25.3  23.1  26.4   2.9
    6. 209.85.254.247                0.0%    10   39.1  39.4  39.1  39.7   0.2
    7. 64.233.174.46                 0.0%    10   39.6  40.4  39.4  46.9   2.3
    8. gw-in-f147.1e100.net          0.0%    10   39.6  40.5  39.5  46.7   2.2

[](javascript:void(0); “Copy Code”)

Rate limiting was previously mentioned. It can cause one or a few intermediate routing nodes to show packet loss or latency, but it does not affect subsequent nodes. ICMP rate limiting means ICMP protocol is only a support/auxiliary protocol used for network probing, not real data traffic. To allocate limited bandwidth more to real, effective data, routing nodes (especially backbone networks) impose traffic limits on ICMP.

Timeout

Timeouts can have many causes. For example, nodes might drop ICMP packets directly or fail to return ICMP response packets, so the report will show “???”.

[](javascript:void(0); “Copy Code”)

root@localhost:~# mtr --report www.google.com
HOST:  localhost                     Loss%   Snt   Last  Avg  Best  Wrst   StDev
    1. 63.247.74.43                  0.0%    10    0.3   0.6   0.3   1.2   0.3
    2. 63.247.64.157                 0.0%    10    0.4   1.0   0.4   6.1   1.8
    3. 209.51.130.213                0.0%    10    0.8   2.7   0.8  19.0   5.7
    4. aix.pr1.atl.google.com        0.0%    10    6.7   6.8   6.7   6.9   0.1
    5. ???                           0.0%    10    7.2   8.3   7.1  16.4   2.9
    6. ???                           0.0%    10   39.1  39.4  39.1  39.7   0.2
    7. 64.233.174.46                 0.0%    10   39.6  40.4  39.4  46.9   2.3
    8. gw-in-f147.1e100.net          0.0%    10   39.6  40.5  39.5  46.7   2.2
```[![](http://assets.cnblogs.com/images/copycode.gif)](javascript:void(0); "Copy Code")

Timeout is not a necessary condition for packet loss, meaning that packet loss does not always have to be accompanied by a timeout. A timeout may occur due to QoS causing the router to discard packets or ICMP response packets failing to return.

Advanced
==

Newer versions of the mtr tool support using TCP packets for probing, but in most cases, such packets should not be used. This is because tcp-based mtr uses SYN packets for probing, and most routing nodes do not respond to such packets.

However, in certain specific scenarios, TCP packets can still be used. When we need to verify certain firewall rules, we can use TCP packets to check protocols or ports—perhaps due to misconfigured port forwarding.

sudo mtr --tcp --port 80 --report --report-cycles 10 speedtest.dallas.linode.com
sudo mtr --tcp --port 22 --report --report-cycles 10 50.116.25.154


Summary
==

This article roughly explains how to use the mtr command, with the main focus and difficulty on understanding the mtr report. However, even with a correct understanding of the report, we may be powerless against some network issues. A packet traverses many network segments from our source to the target host, including the local ISP, intermediate ISPs, and the target ISP. Intermediate ISPs are more likely to involve different countries, so if a problem occurs within an intermediate ISP, there is basically nothing we can do but hope for the best.

What we can do is to try our best to correctly interpret the mtr report and provide the report to the ISP.

Among the advanced features, we can use TCP/UDP protocols to probe specific protocols and ports, which is a functionality that operations personnel may need.

But for all these network-related tools, to use them well, we must have a certain foundation of network knowledge, such as the basics of the TCP/IP protocol.