[URGENT] Ubuntu 10.10 x64, server PC freezes randomly!

[Deleted member 94297]

Hi.
I'm having serious problems with the server I have...
I've installed a server edition of Ubuntu 10.10 x64 edition a few days ago. It freezes randomly. You can operate, if you are IN the server trough SSH already, tried top and nload commands during that time. Everything seems normal. But if I turn off SSH and try to connect trough SFTP or SSH, then it doesn't connect. Like no connection... Which means it's not a connection problem.
I turned on some server, which FPS can be measured, it was almost constant, as it should be during that period of time, which probably rules out the CPU problem.

Maybe I need to configure something for that PC or install something, or do something else? I've installed Linux first time, not counting Mint, so I don't know much about Linux yet...
Here's the configuration:

Intel Processor: Intel® Core™ i3-550, 2x2x3.2Ghz
RAM memory: 4 GB ECC DDRIII
Hard drive: RAID1 2x SATA2 1.5TB 7200RPM
Raid array: Integrated SATA2 RAID levels 0/1/5/10
Disk bay: Hotswap
Server platform: Intel components

Disk /dev/sda: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0008ddef

Device Boot Start End Blocks Id System
/dev/sda1 * 1 180961 1453564928 83 Linux
/dev/sda2 180961 182402 11571201 5 Extended
/dev/sda5 180961 182402 11571200 82 Linux swap / Solaris

Disk /dev/sdb: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000668d5

I've formatted HDD as / 50GB, 700GB /home, 700GB /root and 50GB swap.

I only noticed this because I let my friend host few of his game servers, while I'm making a program on my own PC. Paid for server PC before I did my work, because that was the last one, they are out of stock now, so I'd have to buy another or from other company.

Really need help with this one, as freezes aren't good. And I don't have graphical nor physical access to the PC. I have SSH console and could get a KVM device connected.
Thanks.

 

[Automata]

When it "freezes", does the server work fine from the local terminal (non-network based connection)? It almost sounds like a firewall setup issue on that computer. You could try disabling it temporarily by typing "sudo service iptables stop". You will need to do that from a working SSH connection or from the server itself (preferred).

If it keeps happening, see if the server has a connection to other computers. You will need to actually be at the server, though.

The information you've given isn't exactly related to the issue. I doubt this is hardware related if the server continues running.

 

[Deleted member 94297]

Well, as I said - I don't have physical access to the computer, so I can't test the terminal access, but it would be strange, because it works from SSH access too. At least nload and top commands, don't know any other ones. And cd/dir... That kid of commands.
But how come, that this is happening only occasionally, if it's a firewall problem? I mean, it would or wouldn't let you connect in and out of the server (the game servers uses up/down connection all the time, this also proves, that the server has connection to anywhere. At the time - it should have too, because I'm connected trough SSH), it wouldn't let you connect trough SSH, nor the game servers.
Also, the game servers are frozen, you can't type anything in them, that doesn't seem as an internet issue, because they don't rely on it. You can run one without an internet, except only you would be able to connect.

I compiled my own kernel too. ZEN2 patch and .36 version of it. I changed these settings:

Made all Hard Disk configuration to * instead of M and these:
General setup:
RCU Subsystem (ignore if not present)
Enable RCU Implementation (Preemptible RCU)
Disable Enable tracing for RCU
Processor type and features:
Enable Tickless System (Dynamic Ticks)
Enable High Resolution Timer Support
Select your processor under Processor family
Change Preemtion Mode to Complete Preemption (Real-Time)
Disable Enable tracing for RCU - currently stats in debugfs (ignore if not present)
Enable Machine Check Exception and select Intel or AMD depending on your CPU
Change Timer frequency to 100 HZ
Power management and ACPI options
Enable Power Management support
Disable Power Management Debug Support
Disable Suspend to RAM and standby
Disable Hibernation (aka 'suspend to disk')
Enable ACPI (Advanced Configuration and Power Interface) Support
Disable CPU Frequency scaling
Disable CPU idle PM support
Networking support
Networking options
Enable Packet socket: mmapped IO
Disable QoS and/or fair queueing (Unless you need and use it...)
Device Drivers
Disable Watchdog Timer Support
Enable Real Time Clock
Enable PC-style 'CMOS'
Kernel hacking
Disable everything


Everything else - I left default.

 

[Automata]

Can you boot to a stock (generic) kernel to see if that works? You may have selected an option that makes the system (or its programs) unstable.

If you are connected through SSH, the programs are running fine (top, etc), but no services are available, the server isn't locked up. If it was, nothing would work, even pre-existing connections. This is why I was saying it isn't frozen.

I would try the stock kernel first, to see what happens. If it works, delete the broken kernel and start over.

 

[Deleted member 94297]

Is there any way to boot up from the stock kernel without having physical or KVM access? KVM access is a pain to get, since they have only limited numbers of it... :/

 

[Automata]

You can change the boot order by editing GRUB through nano.

sudo nano /etc/grub.conf

Look for "Default 0". 0 is the index and means the first in the list (which is the same order as the file). If the stock kernel is second in the list, change that zero to a one. If it is third in the list, change that zero to a two. Restart once you get it set right:

sudo shutdown -r now

 

[Deleted member 94297]

Tried the default kernel, it didn't helped the same day. Then the next day everything was okay with no reason.
Today - connection problems for about an hour now. Even getting connection errors with SFTP to the computer.

The server is running on the default configuration, which means that either the default configuration needs to be changed, or there's some other problems... Any ideas? :/
Thansk.

 

[Automata]

If you are paying for the server, I would get their support involved.

 

[dropadrop]

So what is the server running, or is it just sitting idle? There was no cpu usage when it's freezing?
How about with iotop (if it's such a new version that it's included) or "iostat -xk 5".
Do you have a lot of IO during the spikes?
Is the machine swapping?
Do you get errors in logs? How about with the dmesg command?

You know the problem could as well be in the network as the server, something like that is pretty hard to debug. You could set up a cron job to output cpu usage and io stats to a file every now and then, and see if it caught anything. If nothing then do the same for processes, do you have anything new appearing in the list when it freezes? It's for reasons such as this that I prefer to have all servers and network gear monitored with something like cacti or mrtg. When you have a problem you can immediately count out a lot of possibilities.

BTW, in my opinion compiling your own kernel for a server is quite a big no no unless you know what you are doing and why. Then again, I would not go with Ubuntu on a server either. If Ubuntu is your thing then I'd use Debian on the server, otherwise CentOS. There is a reason those distro's don't get all the new stuff so quickly, and while the conservatism makes it less sexy they really do break less. That said, I do run ubuntu at work on my desktop and at home on my htpc (right tool for the job and all that...).