This article is a continuation of the introductory article on ‘Troubleshooting Network Issues’ that was carried in the previous issue of Open Source For You. Those joining us now are advised to read the previous article to understand this one better.
Linux systems support device names based on hardware attributes. This device-naming approach is implemented on physical systems. It requires the biosdevname package to be installed and the biosdevname=1 kernel parameter to be specified. Hardware vendors enable this, by default, on most systems.
On-board network devices are assigned names similar to eno1, where ‘en’ is the Ethernet prefix, and ‘o’ stands for ‘on-board’, followed by a device number. PCI Express hot plug slot devices have names similar to ens1, where ‘s’ stands for ‘slot’. Other network devices have names based on the physical location of the hardware connector enp2s0. Traditional, unpredictable naming, such as eth0, is used if the previous methods do not work.
Note: Network devices of virtual guests use traditional network device naming.
The /etc/udev/rules.d/80-net-name-slot.rules file implements persistent naming. This file is symbolically linked to /dev/null when traditional device-naming is desired. Systems administrators can also write their own udev device-naming rules in /etc/udev/rules.d/70-persistent-net.rules. These rules override the default network device-naming performed by systemd.
Log files play a very important role in troubleshooting network issues. These files provide useful information that helps troubleshoot network device-naming issues. The /var/log/messages file has device detection information that is read at boot time. It also records the messages that the network manager produces when it tries to activate the device. The example in Figure 1 shows the logs related to the ens10 interface in /var/log/messages.
Correcting network device-naming issues
A network interface device’s name can be assigned by setting values in the ifcfg-* configuration files. The MAC address must be matched with the HWADDR variable. The value of the DEVICE variable determines the name of the network interface device corresponding to the specified MAC address. Figure 2 shows the configuration of an interface such as ens10.
The screenshot in Figure 3 shows the configuration of an interface from the command line. The following command can be used to view the configuration of the interface from the command line:
# ip addr show dev <device-name>
Note: Remember that the file name is needed to start with ifcfg-, but it does not have to match the specified device’s name, although that is a recommended practice. Once the changes are made to the configuration file, the system is rebooted to implement them.
Troubleshooting interface configuration problems
Once network device-naming issues have been resolved, there are other network interface configuration settings that can impair network connectivity. Log files do not provide detailed information about the current network configuration. In that case, the same ip addr command mentioned earlier is very helpful, as it displays the active network settings for the network interfaces as shown in Figure 3.
The ip route command displays the current routing table. The default route should be checked to ensure it is the correct gateway address for this subnet. An output of the same command has been shown in Figure 4.
If the Dynamic Host Configuration Protocol (DHCP) is used to configure a network interface, then the network manager launches dhclient to handle that interface’s configuration. Interface network settings, such as the IP address, network mask, the default gateway and DNS servers, are determined by the values provided by the DHCP server. When a network interface uses static addressing, the systems administrator must correctly specify these values.
Inspecting network interface configuration files
Static IPv4 and IPv6 network configuration settings are defined in the /etc/sysconfig/network-scripts/ifcfg-* files. When the network manager is not used, the values in these files determine the persistent settings for network interfaces. In this case, network configuration changes must be made to these files, and then the network should be restarted to apply the changes. A snapshot of the configuration file is given in Figure 2.
The BOOTPROTO variable determines what type of IPv4 configuration is performed on the network interface. A value of ‘none’ or ‘static’ indicates that the IPv4 network is statically configured. The dhcp or bootp values cause dhclient to be used to manage the interface configuration. The variables to set for static IPv4 configuration include IPADDR0 and PREFIX0.
For backwards compatibility, the older IPADDR and NETMASK settings are also supported. When the network interface connects to the external network, the GATEWAY variable should have the IP address of the default gateway.
The IPV6INIT variable must have the value ‘Yes’ for any IPv6 configuration to take place on the network interface. IPV6ADDR specifies the primary static IPv6 address and net mask. Additional IPv6 addresses can be assigned with the IPV6_SECONDARIES variable. The IPV6_AUTOCONF variable determines if IPv6 auto configuration is performed on the network interface.
Managing the network manager configuration settings
When the network manager is used to manage network interfaces on a system, the nmcli command can be used to display and change network settings. nmcli con show ‘CONNECTION_NAME’ is a command that displays the current network configuration for a network manager connection. Values that start with ipv4 or ipv6 correspond to the IPv4 or the IPv6 configuration of an interface, respectively. The Ipv4 and Ipv6 configuration of device ens10 is shown in Figure 5.
The ipv4.method and ipv6.method settings define how IPv4 and IPv6 addressing is configured on the interface. A value that auto indicates DHCP is used for network configuration. When manual is specified for an IP protocol, then IP information must be specified with addresses and gateway settings. The network manager updates the network interface configuration files when nmcli is used to change and assign values to these settings. Another approach would be to change the configuration files in /etc/sysconfig/network-scripts, and then use the command nmcli con reload to have the network manager read them for configuration changes.
The network manager’s network interface configuration changes are not applied to an interface if it is already active. In this case, the interface must be bounced, brought down and back up, to effect the configuration changes. The following commands can be used to bring down the interface and then bring it up again:
# nmcli con down ens10 # nmcli con up ens10
The same has been shown in Figure 6 for device ens10.
Restarting affected network services
Network services can still be unavailable when network interface configuration changes have been made. The affected services should be restarted so they listen to the new address on a changed network interface. Use the following command to restart the network services:
# systemctl restart network
Solving DNS resolution problems
Client-side DNS resolution problems can often cause services to be sluggish or unavailable. For example, the SSH daemon has a UseDNS SSH setting. It determines whether sshd should check if the host name matches the IP address of incoming connections. This feature is enabled, by default, in Linux systems.
The main configuration file for DNS resolution is /etc/resolv.conf. This file identifies the current DNS servers. Figure 7 is a snapshot of /etc/resolv.conf.
/etc/resolv.conf is a good starting point for DNS troubleshooting, this file is often managed by other scripts. The DNS variables defined in the network interface files, /etc/sysconfig/network-scripts/ifcfg-*, can define the name server entries in /etc/resolv.conf. The PEERDNS variable must be set to ‘Yes’ for these settings to be applied. Figure 8 shows how network configuration settings and DNS values are assigned on this host.
Note: The PEERDNS=yes setting also configures dhclient to modify /etc/resolv.conf when using DHCP to assign network values.
When the network manager is active, it makes changes to the network interface configuration files in /etc/sysconfig/network-scripts. In these cases, DNS configuration changes need to be made to the network manager’s configuration information. This is accomplished using the nmcli command. Use the following commands for the same:
# nmcli conn show <device-name> | grep dns // To check the DNS. # nmcli conn mod <device-name> ipv4.dns <dns-value> // The modified DNS that you want to achieve. # nmcli conn show <device-name> | grep dns //To verify the DNS
Although the configuration change has been made to the network manager, /etc/resolv.conf still has its original configuration. The nmcli command must be used to bounce the network connection to apply the configuration changes.
Troubleshooting firewall issues
Once basic network connectivity has been confirmed, a network service may be unavailable because of firewall problems. The firewall rules on the server hosting the network service may have blocked the ports used to access the service. Less frequently, firewall issues can sometimes be a problem on the client machines.
The firewall-cmd command can be used to display and adjust current firewall settings. Invoking the command with the –list-all-zones option gives a complete overview of the firewalld configuration (see Figure 11).
Using the –permanent option with firewall-cmd displays the persistent settings. Comparing active firewalld settings with persistent settings can sometimes identify potential problems.
A quick way to convert runtime rules into permanent rules is with the following command:
# firewall-cmd --runtime-to-permanent
Firewalld is built on top of net filter functionality. This means iptables can be used to get firewall information at a lower level than firewall-cmd.
Similarly, the ip6tables command displays and manages the IPv6 firewall tables.
