Don't feel bad, scaling a website is not easy doubly so when run for the love of it rather than for money as budgets will always be tight. You guys are doing a great job and have a great community.
As an educated guess (bearing in mind I can't see any stats, etc.)!!:
RAM usage is climbing because you have a high level of concurrent Apache processes. This will get worse as the site slows as you end up with more concurrent connections for the same level of traffic. Switching to Apache 2.2 using mpm_worker will help (although isn't really advised with PHP due to some PHP libraries not being thread safe) but nginx with php-fpm will be vastly more efficient in this scenario in terms of RAM usage and CPU.
There's a good guide on getting it working on Ubuntu here: http://www.howtoforge.com/installing-ng ... untu-10.10
The 20% CPU usage, is that reporting user cpu, or combined with system/wait cpu? If it's only user CPU then the bottleneck could be wait io - which is the CPU having to wait for either hard drives or the network. This can also be caused by the server running out of RAM (from Apache starting too many processes for example) and swapping out to disc. Using swap will kill the server stone dead, so that's the doomsday scenario.
If you're serving mostly cached content then the best thing to do is try and bypass PHP completely and serve straight from disc. This removes the huge (relative) overhead of PHP and at least one extra copy operation in RAM between the user space program and the kernel (and probably another copy or two between PHP and Apache). In Apache you'll see some benefit to this, but in nginx you'll see an order of magnitude speed boost.
Ask the ISP to check bandwidth to the servers as well. A few years ago we helped one of the teams out when they had problems with their hosting just before their car launch. It's the only time I've ever seen a server swamp a 100Mbit/s network using only 10% cpu - with all the journalists downloading pics of the cars.
Splitting the website onto two servers (one web, one db) likely won't help as one or other will be the bottleneck for you. But a load balanced 3 server setup would double your throughput. But that'll cost lots and won't be needed for the rest of the year.