All sites suddenly stopped working

[d9000]

At some point last night, all websites hosted on my VPS stopped working. I am able to log in via SSH and have tried rebooting the server through my.vpslink.com, but I can't get lighttpd or lxadmin running.

If I SSH to the server and type:
service lighttpd start

I get the following:
Starting lighttpd: 2008-06-13 19:13:08: (mod_fastcgi.c.900) bind failed for: unix:/tmp/php.socket.[website].com.3458-0 Permission denied
2008-06-13 19:13:08: (mod_fastcgi.c.1336) [ERROR]: spawning fcgi failed.
2008-06-13 19:13:08: (server.c.895) Configuration of plugins failed. Going down.
[ OK ]

The file it is trying to create does not already exist in the /tmp directory, so I have to give full 777 permission so I can write it as root. It appears that I may need to switch to the lxlabs user, but I am getting this:

-bash-3.1# su - lxlabs
su: warning: cannot change directory to /home/lxlabs: Permission denied
su: /sbin/nologin: Permission denied

Anyway - if I change /tmp permissions to 777 and try starting lighttpd, I then get the following errors:

Starting lighttpd: [ OK ]
2008-06-13 19:20:31: (mod_fastcgi.c.990) chdir failed: Permission denied /usr/bin/lxsuexec
2008-06-13 19:20:31: (mod_fastcgi.c.999) execve failed for: /usr/bin/lxsuexec Permission denied
-bash-3.1# 2008-06-13 19:20:31: (mod_fastcgi.c.1025) the fastcgi-backend /usr/bin/lxsuexec failed to start:
2008-06-13 19:20:31: (mod_fastcgi.c.1029) child exited with status 13 /usr/bin/lxsuexec
2008-06-13 19:20:31: (mod_fastcgi.c.1032) If you're trying to run PHP as a FastCGI backend, make sure you're using the FastCGI-enabled version.
You can find out if it is the right one by executing 'php -v' and it should display '(cgi-fcgi)' in the output, NOT '(cgi)' NOR '(cli)'.
For more information, check h_t_t_p://trac.lighttpd.net/trac/wiki/Docs%3AModFastCGI#preparing-php-as-a-fastcgi-programIf]Docs:ModFastCGI - lighttpd - secure, fast, compliant, and very flexible web-server - Trac this is PHP on Gentoo, add 'fastcgi' to the USE flags.
2008-06-13 19:20:31: (mod_fastcgi.c.1336) [ERROR]: spawning fcgi failed.
2008-06-13 19:20:31: (server.c.895) Configuration of plugins failed. Going down.

... and it never seems to actually start up

If I try 'service lxadmin start', I get:

Starting lxadmin: 26466
Could not connect to Mysql server... <br> 2008-06-13 19:22:10: (mod_accesslog.c.535) opening access-log failed: Permission denied /usr/local/lxlabs/lxadmin/log/access_log
2008-06-13 19:22:10: (server.c.895) Configuration of plugins failed. Going down.
mysqld is stopped


These errors all seem to stem from permissions problems, but I am logged in as root. Any ideas on how to switch to a user and group that the server would be happy with?

Thanks

[d9000]

One more thing I noticed... when I change /tmp permissions to 777 and try 'service lighttpd start', it writes php.socket.* files as apache:apache. All other files in that directory are owned by root:root or lxlabs:root.

Where is "apache" coming from? I don't have any apache processes running on the server, and I am logged in as root.

[The Universes]

Did you recently upgrade/update the server software? Sounds like something changed the permissions of a bunch of files needed by lighttpd and it is unable to access them as a result. You should be able to go through and change them back manually.

Do you have Apache installed or anything?

[d9000]

No, I didn't change anything on the server except a LXAdmin update on June 4th.

I posted this on LXLabs forum and got some more clues:
h_t_t_p://forum.lxlabs.com/index.php?t=msg&th=5764&start=0&


It looks like I may have been hacked or the permissions were otherwise massively corrupted across the board. I'm looking at doing a clean reinstall now (actually I'm looking at moving back to shared hosting with a mysql container, as I'm realizing I don't have the stomach or the expertise to manage a VPS).

[The Universes]

A re-install looks like a good place to start. Heres what I'd say, your not going to learn unless you go through the process and yes, the troubles. I can honestly say that my first dedicated server box was hacked in about a month because I knew nothing about security. A lot has changed in 3-4 years and I've learned to lock everything down.

Good luck!