Cloud Firewalls are designed to be intuitive, but the combination of multiple network policies and interaction with software active on Droplets can sometimes lead to unexpected results.
This guide explores strategies to gather information about your network policies and troubleshoot DigitalOcean Cloud Firewall issues. It covers how to:
The first step in diagnosing problems in your Firewall policy is to understand the which rules are being applied. The control panel contains information about individual Firewalls as well as the combined rule set that applies to each Droplet.
To view all of your Firewalls in the DigitalOcean Control Panel, click on the Networking item in the main navigation pane and then select the Firewalls subitem. The Firewall index page lists basic information about each of your Firewalls like the number of Droplets it applies to, the number of rules it contains, and when it was created:
Clicking the name of a Firewall takes you to the Firewall detail page where you can see the policy’s individual rules and the Droplets it targets. This is also where you can add and remove Droplets and tags.
Reviewing the rules applied by each of your Firewalls can provide an overview of the types of network policy being enforced on your infrastructure as a whole.
The Firewall index and detail pages are not the best place to look if you are interested in the policy affecting an individual Droplet. Instead, check the Droplet’s detail page.
From the Droplets item in the main control panel navigation menu, click the Droplet’s name. From there, click Networking in the Droplet menu. Towards the bottom of the page, a section called Firewalls lists the Firewalls that target the Droplet. Below, the policies from each of the Droplet’s Firewalls are combined into a comprehensive table:
These inbound and outbound rules represent all of the traffic allowed through the Firewall for the Droplet. If you are experiencing issues with connectivity to your Droplet after applying a Firewall policy, this is a good place to look.
While the DigitalOcean Cloud Firewall service provides a network-based firewall solution, many administrators may also have a host-based firewall on the Droplet itself. Popular host-based firewalls include IPTables, UFW, and firewalld. When troubleshooting connectivity issues, check these services because they may also be filtering traffic on the Droplet level.
If your server runs a UFW firewall, you can view the current filtering rules by typing:
sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip
To Action From
-- ------ ----
22/tcp (OpenSSH) ALLOW IN Anywhere
22/tcp (OpenSSH (v6)) ALLOW IN Anywhere (v6)
The output shows a few things:
Status
line indicates that the firewall is actively enforcing traffic rules.Default
line indicates that the incoming policy is to deny traffic and the outgoing policy is to allow traffic.If you have a DigitalOcean Cloud Firewall in place with conflicting rules, you can disable your UFW firewall by typing:
sudo ufw disable
If you want to keep UFW in place as a second line of defense, you should modify its rules to match the DigitalOcean Cloud Firewall settings. You can learn how to modify the UFW rules by following the UFW Essentials: Common Firewall Rules and Commands guide.
If your Droplet runs a firewalld firewall, you can view the current traffic rules with a sequence of checks.
First, check whether firewalld is active:
sudo firewall-cmd --state
running
If the firewall is running, check the active zones:
sudo firewall-cmd --get-active-zones
public
interfaces: eth0
If firewalld has active zones, it means that it is evaluating traffic against a set of rules. We can display the ports and services associated with each of the active zones with the --info-zone
option:
sudo firewall-cmd --info-zone=public
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: dhcpv6-client ssh
ports:
protocols:
masquerade: no
forward-ports:
sourceports:
icmp-blocks:
rich rules:
Check whether any ports are open by looking at:
target
value: If this is set to DROP
or %%REJECT%%
, all traffic will be denied regardless of the other settings.ports
value: this lists ports explicitly allowed through the firewallservices
value: this lists services that are allowed through. These are specific sets rules that can be allowed through as a unit.Here, we can see that the dhcpv6-client
and ssh
services are allowed through the firewall. We can check the ports associated with these services by typing:
sudo firewall-cmd --permanent --get-ports --service=dhcpv6-client
sudo firewall-cmd --permanent --get-ports --service=ssh
546/udp
22/tcp
If you have a DigitalOcean Cloud Firewall in place with conflicting rules, you can disable your firewalld firewall by typing:
sudo systemctl stop firewalld
sudo systemctl disable firewalld
If you want to keep firewalld in place as a second line of defense, you should modify its rules to match the DigitalOcean Cloud Firewall settings. You can learn how to modify the firewalld rules by following the How To Set Up a Firewall Using FirewallD on CentOS 7 guide.
If you are using an IPTables firewall, you can view the current IPv4 filtering rules by typing:
sudo iptables --line-numbers -vL
Chain INPUT (policy DROP 1 packets, 40 bytes)
num pkts bytes target prot opt in out source destination
1 0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
2 764 56512 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
3 9 540 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
4 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 374 packets, 97673 bytes)
num pkts bytes target prot opt in out source destination
The output indicates:
policy
for incoming and outgoing traffic (DROP for incoming, and ACCEPT for outgoing)If you are more comfortable with IPTables configuration syntax, you may find the rules-based view more helpful:
sudo iptables -S
-P INPUT DROP
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
The iptables
command only displays the rules for filtering IPv4 traffic. To show the IPv6 filtering rules, rerun the above commands using the ip6tables
instead of iptables
.
If your DigitalOcean Cloud Firewall has rules that conflict with your IPTables firewall, you can disable your IPTables firewall by typing:
sudo iptables -P INPUT ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -F
sudo ip6tables -P INPUT ACCEPT
sudo ip6tables -P OUTPUT ACCEPT
sudo ip6tables -P FORWARD ACCEPT
sudo ip6tables -F
If you are using a service like iptables-persistent
or have a script loading IPTables rules at boot, you may have to disable those as well.
If you want to keep IPTables in place as a second line of defense, you should modify its rules to match the DigitalOcean Cloud Firewall settings. You can learn how to modify the IPTables rules by following the Iptables Essentials: Common Firewall Rules and Commands guide.
If you are experiencing connection problems, it’s useful to verify which services are actually running on your Droplets. Sometimes a service might go down or be under heavy load, which can cause some of the same symptoms as a closed port.
On your Droplets, you can view the services currently bound to a port by typing:
sudo netstat -plunt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 13704/nginx -g daem
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1762/sshd
tcp6 0 0 :::80 :::* LISTEN 13704/nginx -g daem
tcp6 0 0 :::22 :::* LISTEN 1762/sshd
Check the port numbers and interfaces in the Local Address
section against your expected services. If one of your expected services is not present in the netstat
output, your service isn’t running. This would cause connection attempts to fail.
Gathering information about the state of your DigitalOcean Cloud Firewall rules and any host-based firewall rules helps you understand the intended network access. In addition, testing helps you validate the actual traffic filtering behavior.
By installing tools like nmap
on an outside host, we can check which ports we can connect to on our Droplets. If you have Firewall rules that limit access based on the originating host, you may have to install nmap
on multiple machines (those in the allowed group and those outside) for a full understanding.
nmap
on a single test Droplet and then add or remove tags from that server to change which rules match the connection request.For more complete information about scanning hosts with nmap
, follow our guide on How To Test your Firewall Configuration with Nmap and Tcpdump.
To start, you will need to install nmap
on a Droplet that is not a part of the infrastructure you wish to scan.
To install nmap
on an Ubuntu or Debian testing Droplet, type:
sudo apt-get update
sudo apt-get install nmap
To install nmap
on a CentOS testing Droplet, type:
sudo yum install nmap
You should now be able to check the target infrastructure.
Once the scanner is installed, you can scan for open TCP ports by typing:
sudo nmap -sS -Pn -p- -T4 -vv --reason Droplet_IP_address
The scan will attempt a SYN TCP connection to every port on the target Droplet. If you want to limit the number of scanned ports to speed up the scan, you can replace the -p-
option with a range or sequence of ports instead:
sudo nmap -sS -Pn -p 1-5000 -T4 -vv --reason Droplet_IP_address
If you remove the -p
option entirely, nmap
will scan the 1000 most common ports, which is good when you have limited time or are only interested in the state of common service ports:
sudo nmap -sS -Pn -T4 -vv --reason Droplet_IP_address
If you have rules that limit traffic according to origin, run this test against your target infrastructure from a host where connections are supposed to be allowed and again from a host where connections are supposed to be denied.
Because UDP is a connectionless protocol, scanning with UDP is significantly more error prone and time consuming than TCP scans. Since the Cloud Firewalls block traffic by silently dropping offending packets, it is difficult or impossible to distinguish between open and filtered UDP ports by port scanning.
This is beneficial in most situations as it makes it more difficult for outside parties to understand the services running on your infrastructure. However, this interferes with your ability to audit your UDP Firewall rules and so it cannot be used reliably for this purpose.
You can read more about the limitations of UDP scanning in our article on testing firewalls with nmap and tcpdump.
So far, we’ve demonstrated how to gather information about your network policies and audit them. If you are still having trouble tracking down the source of your problems, some of the information in this section may help.
The message you receive when attempting to connect to your server often contains valuable information about the source or type of problem you are experiencing. The exact user-facing message may be different depending on which type of client you are using. For example, SSH clients will have different failure messages than web browsers, and web browsers will each have their own style of messaging.
When DigitalOcean Cloud Firewalls filter traffic, they drop packets that do not match any exception rules. Eventually, after repeated failed attempts, the client connection typically times out. If you see a connection timeout message when attempting to connect to your Droplets, it may be caused by your DigitalOcean Cloud Firewalls rules.
Keep in mind that many other types of scenarios can also cause connection timeouts. For instance, many host-based firewalls also filter by dropping packets. This is also a status you may see if the service you are trying to reach is not operating or if the server itself has gone down. If a connection was redirected incorrectly by a routing component, this could also be the response.
If you see a timeout error, your Cloud Firewalls policy may be responsible. Double-check your rules to verify that the appropriate inbound and outbound traffic is permitted.
If you immediately receive an error message that is not a timeout, the request was blocked by something other than Cloud firewalls. Some of the following messages fall into this category:
These are all indicators that the connection was denied by some other piece of software. For instance, most host-based firewalls include an option to deny traffic with an ICMP rejection message instead of silently dropping packets. Clients that receive these messages are able to error immediately instead of retrying until a timeout limit is reached.
DigitalOcean Cloud Firewalls do not send rejection messages, so if you see one, it means that your request is being blocked somewhere else.
DigitalOcean Cloud Firewalls drop all traffic as a policy. To allow traffic to pass through, you must define exceptions by adding rules.
Because all rules are exceptions to the default drop policy, the rule set is entirely additive. This means that once you allow traffic in one rule, it is impossible to lock it back down in another rule.
This can lead to unexpected results when more than one rule applies to a connection, especially if they come from separate Cloud Firewalls. If the connection matches any rule, it is able to pass through the firewall. You can view the aggregate rules applied to your Droplet by checking out the Networking section on the Droplet’s detail page.
Cloud Firewalls are able to restrict inbound traffic based on the originating IP address and outbound traffic based on the destination IP address. This works well when the secondary party is another server with a static IP address.
However, you may run into trouble if you try to use this functionality to restrict connections to your home connection. Many internet service providers rotate customer IP addresses over time. This means that although a rule limiting connections to your home IP address may work now, it may stop working in the future.
Keep this in mind if you have Cloud Firewall rules that limit connections. If the address or network range is not stable, the rule will not function correctly over time.
Some applications and services require broader network access than the regular listening port. Not accounting for these situations can lead to difficult to debug issues.
Some examples of additional required network access that might not be immediately obvious:
Not accounting for these additional requirements can cause problems, so be sure to check which ports your services are using with sudo netstat -plunt
.
Learning how to diagnose problems with your Firewalls can help you identify the cause of connection problems. To help avoid some common problems with firewalls, consider adopting some of the practices recommended in our How To Organize Your DigitalOcean Cloud Firewalls guide.