Security in a root-server
I just rented a Virtual Private Server (VPS) which allows me to install and run whatever I want as I have root access to the OS.
I'm really happy!! :o)) as until now I just had a shared server allowing me to run only plain HTML and PHP-stuff - still nice, but not as powerful as a root-server.
Security is a big concern when owning a server which is potentially accessible by anybody and is always on, I am therefore writing a kind of small guide about what I did to try to secure it hoping to help other people that want/have to do the same.
I don't expect to do this the "perfect" way, but it is at least as a start better than nothing.
The first hours
Already 3 hours after the first boot I started seeing all of a sudden the following messages pop up in /var/log/messages:
... Apr 10 19:34:58 MYSERVER sshd[9844]: Invalid user administrator from 203.149.62.221 Apr 10 19:35:00 MYSERVER sshd[10552]: Invalid user oracle from 203.149.62.221 Apr 10 19:35:13 MYSERVER sshd[10608]: Invalid user test from 203.149.62.221 ...
Well, I was still installing the main binaries in the OS, so after silently "thanking" 203.149.62.221 for the nice scan s/he performed on my system I quickly...
- changed the listening port of the sshd daemon from the usual 22 to something more esotic (any port between 49152 and 65535)
/etc/ssh/ssd_config by altering the Port parameter. Doing this avoids that a "quick ports scan" (scans which look only for well-known open ports, like 21 for FTP, 22 for SSH, 80 for HTTP, etc..) has an easy time finding them on my server. I didn't have any other ports open.- disabled SSH-access using a password.
/etc/ssh/sshd_config the parameter ChallengeResponseAuthentication to no.
- disabled root-logins.
useradd -m user_itsme
passwd user_itsme - pls. a different one than the one root has otherwise the two hash values will be the same and it might be somehow bad) deploy the public certificate of the user you're using to connect to the server into the authorized_keys file of the user_itsme on the remote server - if in the previous step you used root to connect and login, you can actually copy his authorized_keys-file and change file owner to user_itsme (chown user_itsme authorized_keys) and file permissions (chmod -R 700 ~/.ssh).ssh -p 123456 user_itsme@serverX without entering a password - make sure that works before continuing.user_itsme serverX=/bin/su - root. Again try if that works with sudo su - user_itsme <user_itsme password> and only go on if tjat works./etc/init.d/sshd restart). Now you shouldn't be able anymore to directly login as root, doesn't matter if the certificates you're using are valid or not.
Scan
After having increased the login security I did a scan for open ports on my server:
nmap -p0-65535 <target server IP>
After a while it returned a list of open ports. After that I installed nessus:
nessus-mkcert to generate the certificate.
nessus-adduser to deploy a new nessus user. I chose the standard options.
In both cases nothing interesting was returned - very well.
Open the file /etc/mail/aliases and under the section # Well-known aliases -- these should be filled in! put your email address, e.g.:
root: itsme@myprovider.com operator: itsme@myprovider.com
Now have a look at /etc/ssmtp/ssmtp.conf. Configure it accordingly, e.g.:
# /etc/ssmtp.conf -- a config file for sSMTP sendmail. # The person who gets all mail for userids < 1000 # Make this empty to disable rewriting. root=<target@email.address> mailhub=mail.<myprovider>.com:25 rewriteDomain= hostname=mail.<myprovider>.com:25 UseSTARTTLS=YES AuthUser=<myuserid> AuthPass=<mypassword> FromLineOverride=YES # optional
And, last but not least, configure /etc/ssmtp/revaliases the sender's address. In my case:
root:SERVERX@donotreply.com:mail.<myprovider>.com:25
The above settings might or might not work - depends on the how the mailserver you're trying to connect to is configured. Start with a configuration with your real data and then modify it slowly.
I admin it's not the greatest mail configuration around, but it seem to work.
Logs
I installed logrotate to take care about the system logs that are written in /var/log/ - it will split the logs, compress the old ones and delete the very old ones. Its configuration file is /etc/logrotate.conf and its syntax is very simple. In Gentoo a cron file should be created automatically upon emerging.
As system logger I used syslog-ng. I configured it to have still as default all messages to be written to /var/log/messages, plus some filters to specialize some of them (e.g. with the below configuration the failed logins will be written to messages plus auth.log).
Here the config file I used (/etc/syslog-ng/syslog-ng.conf):
==========================
options {
chain_hostnames(off);
sync(0);
# The default action of syslog-ng 1.6.0 is to log a STATS line
# to the file every 10 minutes. That's pretty ugly after a while.
# Change it to every 12 hours so you get a nice daily update of
# how many messages syslog-ng missed (0).
stats(43200);
};
#2 sources: a global one (src) and a specialized one for kernel-thingies (kernsrc)
source src {
unix-stream("/dev/log" max-connections(256));
internal();
file("/proc/kmsg");
};
source kernsrc { file("/proc/kmsg"); };
#Destinations
destination messages { file("/var/log/messages"); };
destination authlog { file("/var/log/auth.log"); };
destination syslog { file("/var/log/syslog"); };
destination cron { file("/var/log/cron.log"); };
destination daemon { file("/var/log/daemon.log"); };
destination kern { file("/var/log/kern.log"); };
destination lpr { file("/var/log/lpr.log"); };
destination user { file("/var/log/user.log"); };
destination mail { file("/var/log/mail.log"); };
destination mailinfo { file("/var/log/mail.info"); };
destination mailwarn { file("/var/log/mail.warn"); };
destination mailerr { file("/var/log/mail.err"); };
destination newscrit { file("/var/log/news/news.crit"); };
destination newserr { file("/var/log/news/news.err"); };
destination newsnotice { file("/var/log/news/news.notice"); };
#Filters
filter f_authpriv { facility(auth, authpriv); };
filter f_syslog { not facility(authpriv, mail); };
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_kern { facility(kern); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_user { facility(user); };
filter f_info { level(info); };
filter f_warn { level(warn); };
filter f_crit { level(crit); };
filter f_err { level(err); };
#Links between source, filters and destinations
log { source(src); destination(messages); };
log { source(src); destination(console_all); };
log { source(src); filter(f_authpriv); destination(authlog); };
log { source(src); filter(f_syslog); destination(syslog); };
log { source(src); filter(f_cron); destination(cron); };
log { source(src); filter(f_daemon); destination(daemon); };
log { source(kernsrc); filter(f_kern); destination(kern); };
log { source(src); filter(f_lpr); destination(lpr); };
log { source(src); filter(f_mail); destination(mail); };
log { source(src); filter(f_user); destination(user); };
log { source(src); filter(f_mail); filter(f_info); destination(mailinfo); };
log { source(src); filter(f_mail); filter(f_warn); destination(mailwarn); };
log { source(src); filter(f_mail); filter(f_err); destination(mailerr); };
==========================
After organizing how logs got written I had to set up an automatic notification in case that something nasty shows up in them. I used Logcheck, which is contained in Gentoo in the logsentry package.
I configured it in /etc/logcheck/logcheck.sh just by setting my email address and left everything else as default:
SYSADMIN=<myemailaddress>
I tested it with /etc/logcheck/logcheck.sh and after a few seconds I got an email at home. Refine the contents of /etc/logcheck/logcheck.ignore for the strings (e.g. *ciao*) that are supposed to be ignored - be careful not to set just "*" which would make logcheck ignore everything.
To run it automatically I used the usual vixie-cron. I inserted the name of the login user into /etc/cron.deny just to be sure that he doesn't have access to run any cron jobs.
Here there is a guide which explains how to set up new cron jobs.
So, I ran crontab -e and as I wanted to have logcheck to run every 10 minutes I entered:
#Mi HH DD Mo Day of week #START-logsentry/logcheck every 10 minutes 0 * * * * /etc/logcheck/logcheck.sh 10 * * * * /etc/logcheck/logcheck.sh 20 * * * * /etc/logcheck/logcheck.sh 30 * * * * /etc/logcheck/logcheck.sh 40 * * * * /etc/logcheck/logcheck.sh 50 * * * * /etc/logcheck/logcheck.sh #END-logsentry/logcheck every 10 minutes
One last thing I wanted to do was to set in /etc/security/limits.conf the maximum file size for the login user:
youruser hard fsize 10240 #Max file size 10MB
Afterwards I checked if any files were writable by all with...
find / -type f \( -perm -2 \) -exec ls -lg {} \; 2>/dev/null >output.txt
..., did a chmod 750 on the /etc/ssmtp directory and that's it. I am sure that file security is not good, and well, I don't really know much concerning that area.
Firewall
As I don't have another server with multiple network cards available to make it act as a secure firewall, I had to install the firewall directly on this server.
I used iptables as firewall as it is the most used and tested firewall around.
To enable iptables it's best that you modify your kernel configuration and enable EVERYTHING as module under Networking support => Networking options => Network packet filtering framework (Netfilter) in both the Core netfilter configuration and IP: Netfilter configuration.
Be sure to check multiple times in both submenus as some modules pop up in one submenu only when others in the other submenu are selected. I was missing some of them (e.g. the ip_conntrack_ftp module) and iptables was popping up all the time the stupid iptables: No chain/target/match by that name or iptables: Invalid argument error messages and only after several hours of investigating the syntax of my commands I thought that maybe I was missing some modules :o|
!!!Be sure to first test on a local machine what you do next as if you try it out directly on your remote server and the settings are wrong you'll be cut out. You'll have as well to prepare a startup script which executes this, because if you do it in a terminal you'll be cut out right at the beginning because the first commands you execute are the default behavours which will make you disconnect from the terminal!!!
In the end all you need is a script which configures the firewall each time you boot your server.
I created the following one manually - I advice you to do the same to know exactly how the server will react. To write the script I advice you to:
- buy (and read) the book Linux Server Security (or Linux Server-Sicherheit for the german version) written by Michael D. Bauer
- download and install on your local PC Firewall Builder (create a copy of your rules in there - it will help you when you'll get stuck when doing things manually)
- experiment on your local PC and when things don't work check with netstat -tn which ports are being used by whatever got stuck - lsof | grep <searchstring_e.g._ip> will help you too.
Ok, so here is the first version of a script I wrote (mostly taken from Michael D. Bauer s book) for my server (changes will vanish after a reboot - you'll have to write a startup script if you want to activate them automatically):
#<your shell here> #Script iptables_set.sh #REMINDERS-START #List everything #iptables --line-numbers --list #iptables --line-numbers --list INPUT/OUTPUT/FORWARD #Delete a rule #iptables -D OUTPUT 3 #Delete everything #iptables --flush #iptables --delete-chain #iptables -P INPUT ACCEPT #iptables -P OUTPUT ACCEPT #iptables -P FORWARD ACCEPT #REMINDERS-END #============================= #START #Reset everything iptables --flush iptables --delete-chain #Setup the default behaviours iptables -P INPUT DROP iptables -P OUTPUT DROP iptables -P FORWARD DROP #Let everything to the loopback interface iptables -A INPUT -i lo -j ACCEPT iptables -A OUTPUT -o lo -j ACCEPT #Now a "ping localhost" should work #============================= #INPUT-START (From outside world to the server) #REJECT-START #Anti-IP-Spoofing rules iptables -A INPUT -s 255.0.0.0/8 -j LOG --log-prefix "Refused by rule in1\!" iptables -A INPUT -s 255.0.0.0/8 -j DROP iptables -A INPUT -s 0.0.0.0/8 -j LOG --log-prefix "Refused by rule in2\!" iptables -A INPUT -s 0.0.0.0/8 -j DROP iptables -A INPUT -s 127.0.0.0/8 -j LOG --log-prefix "Refused by rule in3\!" iptables -A INPUT -s 127.0.0.0/8 -j DROP iptables -A INPUT -s 192.168.0.0/16 -j LOG --log-prefix "Refused by rule in4\!" iptables -A INPUT -s 192.168.0.0/16 -j DROP iptables -A INPUT -s 172.16.0.0/12 -j LOG --log-prefix "Refused by rule in5\!" iptables -A INPUT -s 172.16.0.0/12 -j DROP iptables -A INPUT -s 10.0.0.0/8 -j LOG --log-prefix "Refused by rule in6\!" iptables -A INPUT -s 10.0.0.0/8 -j DROP #Anti-Stealth rules iptables -A INPUT -p tcp ! --syn -m state --state NEW -j LOG --log-prefix "Refused by rule in8\!" iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP #Auth ident requests iptables -A INPUT -p tcp -m tcp --dport 113 -m state --state NEW -j LOG --log-prefix "Refusd by rule in9\!" iptables -A INPUT -p tcp -m tcp --dport 113 -m state --state NEW -j REJECT #REJECT-END #ACCEPT-START #Traffic that was previously tagged as being ok: iptables -A INPUT -j ACCEPT -m state --state ESTABLISHED,RELATED #SSH new traffic iptables -A INPUT -p tcp -j ACCEPT --dport 22 -m state --state NEW #SSH special port on server iptables -A INPUT -p tcp -j ACCEPT --dport 12345 -m state --state NEW #FTP new traffic #iptables -A INPUT -p tcp -j ACCEPT --dport 21 -m state --state NEW #HTTP new traffic #iptables -A INPUT -p tcp -j ACCEPT --dport 80 -m state --state NEW #PINGs to the server iptables -A INPUT -p icmp -m icmp --icmp-type 8/0 -m state --state NEW -j ACCEPT #Log all the rest - must be the last one iptables -A INPUT -j LOG --log-prefix "Default in-traffic rejection:" #ACCEPT-END #INPUT-END #============================= #OUTPUT-START (From the server to the outside world) #Allow authorized connections - this will make e.g. external SSH connections work iptables -I OUTPUT 1 -m state --state RELATED,ESTABLISHED -j ACCEPT #Allow outgoing SSH connections #iptables -A OUTPUT -p tcp -j ACCEPT --dport 22 -m state --state NEW #SSH special port on server #iptables -A OUTPUT -p tcp -j ACCEPT --dport 12345 -m state --state NEW #Allow outgoing pings iptables -A OUTPUT -p icmp -j ACCEPT --icmp-type echo-request #Allow outgoing DNS requests iptables -A OUTPUT -p udp --dport 53 -m state --state NEW -j ACCEPT #Make "whois" work iptables -A OUTPUT -p tcp -m tcp --dport 43 -m state --state NEW -j ACCEPT #Make "emerge --sync" (rsync) work: iptables -A OUTPUT -p tcp -m tcp --dport 873 -m state --state NEW -j ACCEPT #Make "emerge <whatever>" and HTTP-access work: iptables -A OUTPUT -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT #Make it possible to send out emails: iptables -A OUTPUT -p tcp -m tcp --dport 25 -m state --state NEW -j ACCEPT #Make it possible to check for incoming emails: #iptables -A OUTPUT -p tcp -m tcp --dport 110 -m state --state NEW -j ACCEPT #Log all the rest - must be the last one iptables -A OUTPUT -j LOG --log-prefix "Default out-traffic rejection" #OUTPUT-END
You see that I commented out some commands (like the incoming HTTP requests on port 80) as I don't have (yet) such services running.
Once you reviewed, modified and successfully tested the script on your local LAN, upload it to the server and prepare the root-user to execute it (non-root users won't be allowded to change firewall settings).
Remember that you cannot execute the single steps of the scripts one-by-one as after setting the default behaviours (which will reject EVERYTHING) you'll be cut out. It is therefore very important that you execute the script as a whole AND that you have a "rescue"-strategy in place if anything goes wrong.
One of the "rescue-strategies" is for sure the possibility to remotely restart your server using the external command console of your provider. I do have such a console but didn't want to restart the server each time (which takes quite some time), so I wrote this script...
#<your shell here> #Script runme.sh /home/myrootuser/iptables_set.sh > output_iptables.txt sleep 120 #Undo everything iptables --flush iptables --delete-chain iptables -P INPUT ACCEPT iptables -P OUTPUT ACCEPT iptables -P FORWARD ACCEPT
...which first runs the commands needed to set up the firewall and then gives me 2 minutes to see if I am still able to connect. When the 2 minutes elapse the firewall is set to let everything through and you should get back connectivity if you were previously cut out.
Once you are ready you'll have to run the above script as "nohup ./runme & > output.txt".
"&" will run the script in the background, "nohup" will continue to run the script even if you disconnect from the terminal, and ">" will output all (hopefully none) errors to the file "output.txt".
Once you successfully installed the firewall script on your server and have disabled port 22 for incoming connections when running the command...
nmap -sS -O <IP_of_your_server>
...you should get something like:
~ # nmap -sS -O <IP_of_your_server> Starting Nmap 4.76 ( http://nmap.org ) at 2009-04-13 19:35 CEST All 1000 scanned ports on YOURHOST (123.123.123.123) are filtered Too many fingerprints match this host to give specific OS details OS detection performed. Please report any incorrect results at http://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 26.82 seconds
So, now it is time for you to set the firewall script in your startup scripts, reboot the server and keep fingers crossed. After the server has rebooted check (if you can still connect to it) with iptables --line-numbers --list if everything is still in place.
One last thing: if you're using the Bash shell, write the following into the ~/.bash_logout-file of your users to clear their history when they logout so that no traces which might eventually help an intruder knowing what you last did are left:
history -c echo ciao > ~/.bash_history
Notes
- On Gentoo run from time to time
emerge --syncfollowed by "glsa-check -l affected" to see the list of available security updates. Install them if anything pops up.
References
http://www.gentoo.org/doc/en/security/security-handbook.xml?full=1