F1technical server situation (speed/availability)

Everything about this website and its content. Here you will find update announcements or requests for feedback. Questions about layout, functionality, content, and your suggestions are welcome.
myurr
myurr
30
Joined: Thu Mar 20, 2008 8:58 pm

F1technical server situation (speed/availability)

Post

Quick suggestion... Ditch Apache - move to a lighter weight web server such as nginx as you'll get better utilisation of existing hardware, especially under concurrent loads. Fastcgi based PHP is actually more efficient than compiling it into Apache. This will help with days like today where the load spikes must be quite horrid to deal with.

I don't know phpBB at all, but on a read heavy site like this caching is your friend. MySQL can also have issues scaling in highly concurrent situations particularly with MyISAM tables - but again I don't know how easy it'll be to move phpBB to another architecture. PostgreSQL, in my experience, is a no brainer on more complex sites but can be faster than MySQL when working with large data sets and/or highly concurrent environments. MySQL 5+ with query caching may also be worth benchmarking to see if it helps. Where PostgreSQL (and innodb) struggle is with paging as counting the number of rows in a table means running the full query - MyISAM caches row counts allowing SELECT COUNT(*) FROM ... type queries to be accelerated. MVCC used by Postgres and innodb don't allow for that caching, but you could use triggers to manually maintain a count and get similarly fast lookups.

It's going to be a long shot as this would involve some big tech upheavals, but it would be great to see a real time forum. I don't think I've seen anyone implement one yet, but using a 'comet' server architecture you could in theory organise a forum more like a real time chat. I could see a simplified 'reply' box appearing at the top of each page with a live stream of posts underneath in reverse chronological order. Whilst watching a given thread new posts would dynamically arrive in real time at the head of the column, highlighting for a few seconds to show what's happened. This would happen whilst you were composing a reply so that you could see and react to other posts.

Possibly a worse layout for periodic and casual readers, but it would be better on days like today to have such a stream, and it would be better for frequent posters who are trying to hold a conversation.

The same tech would also allow a real time list of other people logged in and reading a particular thread. It could also be used to implement a better version of James Allen's twitter aggregator - a great service, poorly executed in my view. I'd like to be able to personalise which twitter feeds are displayed to me, and have a real time update rather than a page refresh that loses my scroll position (as currently happens).

I know how to do all the above so know it's possible, but admit it's a little bit on the bleeding edge of things, not for everyone, and would imagine it'd be difficult given the current setup.

User avatar
Steven
Owner
Joined: Mon Aug 19, 2002 5:32 pm
Location: Belgium

Re: F1technical server situation (speed/availability)

Post

Thank for the suggestions!

The problem however at the moment is something weird. I've already sent a few angry mails to hosting company, hoping they will have a deeper look into it.

Right now, server CPU is only used about 20%, so it's kind of lazily hanging around. Memory is quite full, but more importantly the problems appear to be generated by simply retrieving static images. PHP parsing is quick, and DB queries are quick as well.

Yesterday, during the afternoon some cache increases to mysql solved our issues, but the even higher load today is again causing troubles. Note that yesterday we recorded 20% higher pageviews than the best day ever, and today, despite the problems, it's going to be even more.

It's sad though, cause I can't put online much more, as I fear more visitors will cause even more problems. I hope first a solution is worked out pretty quickly.

I'm terribly sorry for this, I feel like it's another missed chance for f1t. Sorry again :(

segedunum
segedunum
0
Joined: Tue Apr 03, 2007 12:49 pm

Re: F1technical server situation (speed/availability)

Post

Yep, nginx and php-fpm will help you out massively. Once a site is heavily loaded Apache has a horrible tendency to make it even worse and mod_php consumes a ton of memory. Completely worth the move. You'll get a lot more out of the same hardware.

myurr
myurr
30
Joined: Thu Mar 20, 2008 8:58 pm

Re: F1technical server situation (speed/availability)

Post

Don't feel bad, scaling a website is not easy doubly so when run for the love of it rather than for money as budgets will always be tight. You guys are doing a great job and have a great community.

As an educated guess (bearing in mind I can't see any stats, etc.)!!:

RAM usage is climbing because you have a high level of concurrent Apache processes. This will get worse as the site slows as you end up with more concurrent connections for the same level of traffic. Switching to Apache 2.2 using mpm_worker will help (although isn't really advised with PHP due to some PHP libraries not being thread safe) but nginx with php-fpm will be vastly more efficient in this scenario in terms of RAM usage and CPU.

There's a good guide on getting it working on Ubuntu here: http://www.howtoforge.com/installing-ng ... untu-10.10

The 20% CPU usage, is that reporting user cpu, or combined with system/wait cpu? If it's only user CPU then the bottleneck could be wait io - which is the CPU having to wait for either hard drives or the network. This can also be caused by the server running out of RAM (from Apache starting too many processes for example) and swapping out to disc. Using swap will kill the server stone dead, so that's the doomsday scenario.

If you're serving mostly cached content then the best thing to do is try and bypass PHP completely and serve straight from disc. This removes the huge (relative) overhead of PHP and at least one extra copy operation in RAM between the user space program and the kernel (and probably another copy or two between PHP and Apache). In Apache you'll see some benefit to this, but in nginx you'll see an order of magnitude speed boost.

Ask the ISP to check bandwidth to the servers as well. A few years ago we helped one of the teams out when they had problems with their hosting just before their car launch. It's the only time I've ever seen a server swamp a 100Mbit/s network using only 10% cpu - with all the journalists downloading pics of the cars.

Splitting the website onto two servers (one web, one db) likely won't help as one or other will be the bottleneck for you. But a load balanced 3 server setup would double your throughput. But that'll cost lots and won't be needed for the rest of the year.

User avatar
WhiteBlue
124
Joined: Mon Apr 14, 2008 7:58 pm
Location: WhiteBlue Country

Re: F1technical server situation (speed/availability)

Post

Just as feed back. The forum is practically unusable today. I was getting reaction times between seconds and one minute which is completely unacceptable from a user's point of view. I hope you find the bugs and get this resolved.
Formula One's fundamental ethos is about success coming to those with the most ingenious engineering and best .............................. organization, not to those with the biggest budget. (Dave Richards)

User avatar
Steven
Owner
Joined: Mon Aug 19, 2002 5:32 pm
Location: Belgium

Re: F1technical server situation (speed/availability)

Post

WhiteBlue wrote:Just as feed back. The forum is practically unusable today. I was getting reaction times between seconds and one minute which is completely unacceptable from a user's point of view. I hope you find the bugs and get this resolved.
Don't rub it in please. I've been unable to do anything here since 5 hours. Not a single carpage, news, nothing. Just been busying myself with phonecalls, twitter and being annoyed. I really am doing what I can, but my day is ruined as well.

andrew
andrew
0
Joined: Tue Feb 16, 2010 2:08 pm
Location: Aberdeen, Scotland - WhiteBlue Country (not the region)

Re: F1technical server situation (speed/availability)

Post

WhiteBlue wrote:Just as feed back. The forum is practically unusable today. I was getting reaction times between seconds and one minute which is completely unacceptable from a user's point of view. I hope you find the bugs and get this resolved.
Bit harsh there. Sure the site is slow but to say it is unacceptable is, well unacceptable. It just means we all have to wait a little longer to impart our wisdom for the day on all things F1.

Here's a thought though. Have a retro day and pretend you're using dial-up of the late 90's. :lol:

Apart from the site speed everything else I have used on here seem to be working as normal.

andrew
andrew
0
Joined: Tue Feb 16, 2010 2:08 pm
Location: Aberdeen, Scotland - WhiteBlue Country (not the region)

Re: F1technical server situation (speed/availability)

Post

Further to my previous post, the sitre seems to be back upto normal speed again.

Cheers.

User avatar
Steven
Owner
Joined: Mon Aug 19, 2002 5:32 pm
Location: Belgium

Re: F1technical server situation (speed/availability)

Post

To me it's still a bit slow, but obviously speeds will improve as less users are online.
Viewing an image in the gallery for instance still takes more than 10s for me right now (of which are 8.7s parsing time on the server)

We are however moving to a different server to see what might be the problem. I'd rather try to prevent anything like this happening again, like for instance during the upcoming McLaren launch... although unfortunately we will probably only know when we get there. Cross your fingers!

donskar
donskar
24
Joined: Sat Feb 03, 2007 3:41 pm
Location: Cardboard box, end of Boulevard of Broken Dreams

Re: F1technical server situation (speed/availability)

Post

Tomba wrote:To me it's still a bit slow, but obviously speeds will improve as less users are online.
Viewing an image in the gallery for instance still takes more than 10s for me right now (of which are 8.7s parsing time on the server)

We are however moving to a different server to see what might be the problem. I'd rather try to prevent anything like this happening again, like for instance during the upcoming McLaren launch... although unfortunately we will probably only know when we get there. Cross your fingers!
For the great service/content you provide, we can put up with some minor delays. Great job 364 days of the year. You get a pass today.
Enzo Ferrari was a great man. But he was not a good man. -- Phil Hill

User avatar
WhiteBlue
124
Joined: Mon Apr 14, 2008 7:58 pm
Location: WhiteBlue Country

Re: F1technical server situation (speed/availability)

Post

It is much faster now. Good luck with all the things you try. I'll keep the fingers crossed.
Formula One's fundamental ethos is about success coming to those with the most ingenious engineering and best .............................. organization, not to those with the biggest budget. (Dave Richards)

marcush.
marcush.
268
Joined: Tue Mar 09, 2004 3:55 pm

Re: F1technical server situation (speed/availability)

Post

on one of these days we suddenly realise what a jewel you are providing to us f1 petrolheads.
Many ,many thanks for this.It is well appreciated.
I´m sure F1t is coming out of this even stronger.

User avatar
samstre
0
Joined: Wed Apr 16, 2008 9:41 pm
Location: Graz [Austria]

Re: F1technical server situation (speed/availability)

Post

All I want to say is:

Thank you! You make an awesome job running this forum! I've been visiting this forum almost daily and the site was never down!

I'm working as a systemengineer. We are hosting Intranet/PM/AM sites for various companies (200-1000 concurrent users). It's quite normal that a webservers performance goes down, as the number of concurrent users increases! You could tune it a bit by tuning MySQL and switching from mod_php to fastcgi. But in the end you'll need more performance (RAM, CPU-Power, Network). You could benchmark your Sites performance with Apache JMeter (http://jakarta.apache.org/jmeter/) - Maybe you can find out what's causing the performance issues!

If you are running PHP you could try to disable open_basedir! They had (or have - depending on the version) a bug thats causing a lot of file stats. I had this problem recently with FreeBSD 8.1, Apache 2.2 and PHP 4.4.9! PHP 5.2 was also affected. (With open_basedir we had ~50000 file stats, without we had 50)

Anyway... Thanks for doing such a great job!
And to the people who can't wait for 10 minutes... Go get a job! *ggg*

Greetz

User avatar
Steven
Owner
Joined: Mon Aug 19, 2002 5:32 pm
Location: Belgium

Re: F1technical server situation (speed/availability)

Post

Even though lots of people online, things look ok now...
Post here if they aren't, I can't follow-up today like yesterday, so that way I have a record somehow.

Looks like moving servers really worked out! (I hope, sincerely).

carvetia
carvetia
0
Joined: Tue Apr 22, 2008 9:51 am

Re: F1technical server situation (speed/availability)

Post

@Tomba

The site is faster for me now than it ever has been (even prior to the peak) so well done 8) On the other hand I did get a memory allocation error yesterday which is a pretty terminal problem, and something that could most likely be avoided without throwing more hardware at the problem, just utilising what you already have more effectively. I concur with the suggestions made above.

@myurr

I've actually been working on something similar to what you suggest for a while that's going to be pulling F1 data from a lot of different sources (this forum, feeds, live timings once they're back up and I can work on parsing them again, galleries etc).

When it's up and ready for testing I'll be authenticating against this forum so users here can test it out. It will let you interact in real-time; it's going to be quite experimental/cutting-edge (web sockets) but hopefully interesting and useful for those who are willing/able to use a modern browser :)