My firewalls are not working correctly
Cloud Firewalls are designed to be intuitive, but the combination of multiple network policies and interaction with software active on Droplets can sometimes lead to unexpected results.
This guide explores strategies to gather information about your network policies and troubleshoot DigitalOcean Cloud Firewall issues. It covers how to:
- Find your active Firewalls
- View the comprehensive set of rules applied to a single Droplet
- Discover host-based firewalls that might be active on Droplet
- Mitigate or resolve conflicting policies
Finding Firewall Information
The first step in diagnosing problems in your Firewall policy is to understand the which rules are being applied. The control panel contains information about individual Firewalls as well as the combined rule set that applies to each Droplet.
Displaying Information About All Firewalls
To view all of your Firewalls in the DigitalOcean Control Panel, click on the Networking item in the main navigation pane and then select the Firewalls subitem. The Firewall index page lists basic information about each of your Firewalls like the number of Droplets it applies to, the number of rules it contains, and when it was created:
Clicking the name of a Firewall takes you to the Firewall detail page where you can see the policy’s individual rules and the Droplets it targets. This is also where you can add and remove Droplets and tags.
Reviewing the rules applied by each of your Firewalls can provide an overview of the types of network policy being enforced on your infrastructure as a whole.
Displaying the Firewall Policy Affecting an Individual Droplet
The Firewall index and detail pages are not the best place to look if you are interested in the policy affecting an individual Droplet. Instead, check the Droplet’s detail page.
From the Droplets item in the main control panel navigation menu, click the Droplet’s name. From there, click Networking in the Droplet menu. Towards the bottom of the page, a section called Firewalls lists the Firewalls that target the Droplet. Below, the policies from each of the Droplet’s Firewalls are combined into a comprehensive table:
These inbound and outbound rules represent all of the traffic allowed through the Firewall for the Droplet. If you are experiencing issues with connectivity to your Droplet after applying a Firewall policy, this is a good place to look.
Displaying Host-Based Firewall Rules Active on Droplets
While the DigitalOcean Cloud Firewall service provides a network-based firewall solution, many administrators may also have a host-based firewall on the Droplet itself. Popular host-based firewalls include IPTables, UFW, and firewalld. When troubleshooting connectivity issues, check these services because they may also be filtering traffic on the Droplet level.
Finding the UFW Firewall Settings
If your server runs a UFW firewall, you can view the current filtering rules by typing:
sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip
To Action From
-- ------ ----
22/tcp (OpenSSH) ALLOW IN Anywhere
22/tcp (OpenSSH (v6)) ALLOW IN Anywhere (v6)
The output shows a few things:
- The
Statusline indicates that the firewall is actively enforcing traffic rules. - The
Defaultline indicates that the incoming policy is to deny traffic and the outgoing policy is to allow traffic. - The rule set at the bottom indicates that incoming connections to port 22 are allowed on both IPv4 and IPv6 addresses.
If you have a DigitalOcean Cloud Firewall in place with conflicting rules, you can disable your UFW firewall by typing:
sudo ufw disable
If you want to keep UFW in place as a second line of defense, you should modify its rules to match the DigitalOcean Cloud Firewall settings. You can learn how to modify the UFW rules by following the UFW Essentials: Common Firewall Rules and Commands guide.
Finding the Firewalld Firewall Settings
If your Droplet runs a firewalld firewall, you can view the current traffic rules with a sequence of checks.
First, check whether firewalld is active:
sudo firewall-cmd --state
running
If the firewall is running, check the active zones:
sudo firewall-cmd --get-active-zones
public
interfaces: eth0
If firewalld has active zones, it means that it is evaluating traffic against a set of rules. We can display the ports and services associated with each of the active zones with the --info-zone option:
sudo firewall-cmd --info-zone=public
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: dhcpv6-client ssh
ports:
protocols:
masquerade: no
forward-ports:
sourceports:
icmp-blocks:
rich rules:
Check whether any ports are open by looking at:
- The
targetvalue: If this is set toDROPor%%REJECT%%, all traffic will be denied regardless of the other settings. - The
portsvalue: this lists ports explicitly allowed through the firewall - The
servicesvalue: this lists services that are allowed through. These are specific sets rules that can be allowed through as a unit.
Here, we can see that the dhcpv6-client and ssh services are allowed through the firewall. We can check the ports associated with these services by typing:
sudo firewall-cmd --permanent --get-ports --service=dhcpv6-client
sudo firewall-cmd --permanent --get-ports --service=ssh
546/udp
22/tcp
If you have a DigitalOcean Cloud Firewall in place with conflicting rules, you can disable your firewalld firewall by typing:
sudo systemctl stop firewalld
sudo systemctl disable firewalld
If you want to keep firewalld in place as a second line of defense, you should modify its rules to match the DigitalOcean Cloud Firewall settings. You can learn how to modify the firewalld rules by following the How To Set Up a Firewall Using FirewallD on CentOS 7 guide.
Finding the IPTables Firewall Settings
If you are using an IPTables firewall, you can view the current IPv4 filtering rules by typing:
sudo iptables --line-numbers -vL
Chain INPUT (policy DROP 1 packets, 40 bytes)
num pkts bytes target prot opt in out source destination
1 0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
2 764 56512 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
3 9 540 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
4 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 374 packets, 97673 bytes)
num pkts bytes target prot opt in out source destination
The output indicates:
- The default
policyfor incoming and outgoing traffic (DROP for incoming, and ACCEPT for outgoing) - The exceptions to the default policies. Here, we can see exemptions for incoming TCP traffic to ports 22 and 80 (as well as traffic related to existing connections and traffic coming from the local interface)
If you are more comfortable with IPTables configuration syntax, you may find the rules-based view more helpful:
sudo iptables -S
-P INPUT DROP
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
The iptables command only displays the rules for filtering IPv4 traffic. To show the IPv6 filtering rules, rerun the above commands using the ip6tables instead of iptables.
If your DigitalOcean Cloud Firewall has rules that conflict with your IPTables firewall, you can disable your IPTables firewall by typing:
sudo iptables -P INPUT ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -F
sudo ip6tables -P INPUT ACCEPT
sudo ip6tables -P OUTPUT ACCEPT
sudo ip6tables -P FORWARD ACCEPT
sudo ip6tables -F
If you are using a service like iptables-persistent or have a script loading IPTables rules at boot, you may have to disable those as well.
If you want to keep IPTables in place as a second line of defense, you should modify its rules to match the DigitalOcean Cloud Firewall settings. You can learn how to modify the IPTables rules by following the Iptables Essentials: Common Firewall Rules and Commands guide.
Viewing Running Services
If you are experiencing connection problems, it’s useful to verify which services are actually running on your Droplets. Sometimes a service might go down or be under heavy load, which can cause some of the same symptoms as a closed port.
On your Droplets, you can view the services currently bound to a port by typing:
sudo netstat -plunt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 13704/nginx -g daem
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1762/sshd
tcp6 0 0 :::80 :::* LISTEN 13704/nginx -g daem
tcp6 0 0 :::22 :::* LISTEN 1762/sshd
Check the port numbers and interfaces in the Local Address section against your expected services. If one of your expected services is not present in the netstat output, your service isn’t running. This would cause connection attempts to fail.
Evaluating Connectivity From an External Source
Gathering information about the state of your DigitalOcean Cloud Firewall rules and any host-based firewall rules helps you understand the intended network access. In addition, testing helps you validate the actual traffic filtering behavior.
By installing tools like nmap on an outside host, we can check which ports we can connect to on our Droplets. If you have Firewall rules that limit access based on the originating host, you may have to install nmap on multiple machines (those in the allowed group and those outside) for a full understanding.
Note
nmap on a single test Droplet and then add or remove tags from that server to change which rules match the connection request.
For more complete information about scanning hosts with nmap, follow our guide on How To Test your Firewall Configuration with Nmap and Tcpdump.
Installing Nmap
To start, you will need to install nmap on a Droplet that is not a part of the infrastructure you wish to scan.
To install nmap on an Ubuntu or Debian testing Droplet, type:
sudo apt-get update
sudo apt-get install nmap
To install nmap on a CentOS testing Droplet, type:
sudo yum install nmap
You should now be able to check the target infrastructure.
Scanning TCP Ports
Once the scanner is installed, you can scan for open TCP ports by typing:
sudo nmap -sS -Pn -p- -T4 -vv --reason Droplet_IP_address
The scan will attempt a SYN TCP connection to every port on the target Droplet. If you want to limit the number of scanned ports to speed up the scan, you can replace the -p- option with a range or sequence of ports instead:
sudo nmap -sS -Pn -p 1-5000 -T4 -vv --reason Droplet_IP_address
If you remove the -p option entirely, nmap will scan the 1000 most common ports, which is good when you have limited time or are only interested in the state of common service ports:
sudo nmap -sS -Pn -T4 -vv --reason Droplet_IP_address
If you have rules that limit traffic according to origin, run this test against your target infrastructure from a host where connections are supposed to be allowed and again from a host where connections are supposed to be denied.
Scanning UDP Ports
Because UDP is a connectionless protocol, scanning with UDP is significantly more error prone and time consuming than TCP scans. Since the Cloud Firewalls block traffic by silently dropping offending packets, it is difficult or impossible to distinguish between open and filtered UDP ports by port scanning.
This is beneficial in most situations as it makes it more difficult for outside parties to understand the services running on your infrastructure. However, this interferes with your ability to audit your UDP Firewall rules and so it cannot be used reliably for this purpose.
You can read more about the limitations of UDP scanning in our article on testing firewalls with nmap and tcpdump.
General Troubleshooting Information
So far, we’ve demonstrated how to gather information about your network policies and audit them. If you are still having trouble tracking down the source of your problems, some of the information in this section may help.
Connections Blocked by DigitalOcean Cloud Firewalls Will Timeout
The message you receive when attempting to connect to your server often contains valuable information about the source or type of problem you are experiencing. The exact user-facing message may be different depending on which type of client you are using. For example, SSH clients will have different failure messages than web browsers, and web browsers will each have their own style of messaging.
When DigitalOcean Cloud Firewalls filter traffic, they drop packets that do not match any exception rules. Eventually, after repeated failed attempts, the client connection typically times out. If you see a connection timeout message when attempting to connect to your Droplets, it may be caused by your DigitalOcean Cloud Firewalls rules.
Keep in mind that many other types of scenarios can also cause connection timeouts. For instance, many host-based firewalls also filter by dropping packets. This is also a status you may see if the service you are trying to reach is not operating or if the server itself has gone down. If a connection was redirected incorrectly by a routing component, this could also be the response.
If you see a timeout error, your Cloud Firewalls policy may be responsible. Double-check your rules to verify that the appropriate inbound and outbound traffic is permitted.
If you immediately receive an error message that is not a timeout, the request was blocked by something other than Cloud firewalls. Some of the following messages fall into this category:
- connection refused
- destination unreachable
- port unreachable
- host unreachable
- net unreachable
- protocol unreachable
- net prohibited
- host prohibited
- admin prohibited
- no route
- address unreachable
- TCP reset
These are all indicators that the connection was denied by some other piece of software. For instance, most host-based firewalls include an option to deny traffic with an ICMP rejection message instead of silently dropping packets. Clients that receive these messages are able to error immediately instead of retrying until a timeout limit is reached.
DigitalOcean Cloud Firewalls do not send rejection messages, so if you see one, it means that your request is being blocked somewhere else.
DigitalOcean Cloud Firewall Rules are Additive
DigitalOcean Cloud Firewalls drop all traffic as a policy. To allow traffic to pass through, you must define exceptions by adding rules.
Because all rules are exceptions to the default drop policy, the rule set is entirely additive. This means that once you allow traffic in one rule, it is impossible to lock it back down in another rule.
This can lead to unexpected results when more than one rule applies to a connection, especially if they come from separate Cloud Firewalls. If the connection matches any rule, it is able to pass through the firewall. You can view the aggregate rules applied to your Droplet by checking out the Networking section on the Droplet’s detail page.
Restricting Connections by Source or Destination Requires Static IP Addresses
Cloud Firewalls are able to restrict inbound traffic based on the originating IP address and outbound traffic based on the destination IP address. This works well when the secondary party is another server with a static IP address.
However, you may run into trouble if you try to use this functionality to restrict connections to your home connection. Many internet service providers rotate customer IP addresses over time. This means that although a rule limiting connections to your home IP address may work now, it may stop working in the future.
Keep this in mind if you have Cloud Firewall rules that limit connections. If the address or network range is not stable, the rule will not function correctly over time.
Applications Sometimes Require Additional Network Access
Some applications and services require broader network access than the regular listening port. Not accounting for these situations can lead to difficult to debug issues.
Some examples of additional required network access that might not be immediately obvious:
- DigitalOcean Monitoring Agent: The DigitalOcean Monitoring agent requires outbound TCP access on ports 80 and 443 to authenticate to the Monitoring service and to send metrics. If you are restricting outbound traffic on these ports, DigitalOcean Monitoring will no longer work.
- DNS Lookups: Many applications require DNS access to function correctly. Blocking outbound DNS (TCP and UDP port 53) may lead to many difficult to diagnose issues.
- Service Coordination: Many applications use additional ports to coordinate with other servers or to accept special types of traffic. For instance, databases use designated ports for clustering or replication.
Not accounting for these additional requirements can cause problems, so be sure to check which ports your services are using with sudo netstat -plunt.
More Information
Learning how to diagnose problems with your Firewalls can help you identify the cause of connection problems. To help avoid some common problems with firewalls, consider adopting some of the practices recommended in our How To Organize Your DigitalOcean Cloud Firewalls guide.