tag:blogger.com,1999:blog-288858287590827405.post2450223985612537026..comments2008-12-28T18:51:34.717-08:00Leftyhttp://www.blogger.com/profile/14637607058852159739noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-288858287590827405.post-11712261160082913362008-12-28T18:51:00.000-08:002008-12-28T18:51:00.000-08:00@Greg - thanks again for the comments. We definitely aren't getting the full picture on our virtual servers. I've looked at rails caching (your link), memcache, and MySQL caching so far - but have not yet implemented it. We'll stay on top of that and I'll report what we find on the blog. <BR/><BR/>For benchmarking we had been using requests/second, not time to load. I thought that was more telling.<BR/><BR/>Without caching, as you said, we're dead in the water. And since we're done with functionality now, we're moving on to optimizing our code. And that includes all this fun caching stuff!Leftyhttp://www.blogger.com/profile/14637607058852159739noreply@blogger.comtag:blogger.com,1999:blog-288858287590827405.post-77179818919500347122008-12-27T11:35:00.000-08:002008-12-27T11:35:00.000-08:00Left,<BR/><BR/>Good thoughts.<BR/><BR/>One thing to remember when you look at your benchmark results is that you are running in a virtual hosting environment. Depending on how visualizations is done, the performance might be much poor than equivalent stand-alone server.<BR/><BR/>I worked on applications written Java as well as C#, and all of them suffered from the same exact issue you are facing now. Nowadays memory is cheap, so adding memory caching substantially increasing your application performance. Just like Java or .NET frameworks, rails provide ways to cache parts of your applications (see the link in my previous post).<BR/><BR/>I don't know the nature of your application, however based on your description it sounds like the bottleneck is in database reads, not writes. If that's the case, then database reads (queries) are directly reflected by how many requests your rails applications processes, which is proportional to the amount of visitors hitting your site (I agree, they don't have to be unique visitors).<BR/><BR/>When you run your benchmarking tests, aside from measuring time to process a single request, I would also measure how many requests a second your application can sustain.<BR/><BR/>Bottom line, reading from disk is the most expensive operation. To minimize it, you have to use system memory to cache disk data. You can do in the application layer (rails, .net, java, etc), or in the database layer(mysql can cash your entire db if required) or both. Without caching, your application won't scale.Genehttp://www.blogger.com/profile/05443943086571159698noreply@blogger.comtag:blogger.com,1999:blog-288858287590827405.post-26062107482212818372008-12-26T13:02:00.000-08:002008-12-26T13:02:00.000-08:00@Gene - Thanks for the comments. You are right in that we haven't implemented much in terms of optimization. While I know there is much that can be done and we will do, I was surprised at the benchmarking scores we measured. I would have expected performance to have been a bit better. We did a real basic benchmark of several of our dynamic pages vs. our static content (load).<BR/><BR/>I do share your opinion in that we'll be able to scale. The issue is more at what expense? Rails, as a language, hits the database more than anything I've seen before. Since the database layer is partially hidden (meaning Rails handles it), it's more difficult to actually tweak what is happening. With Java, my last programming language to play with, you controlled all aspects of the development, so you could easily tell where you could tweak/improve the code. <BR/><BR/>Finally, in terms of Twitter vs. Yellowpages, I'd think that unique visitors a month is irrelevant. Twitter users use the system much more, so their uniques wouldn't really factor into performance issues. There are quite a few large scale sites working on Rails. But our application is incredibly intensive in terms of the data we have to manipulate. Also - Yellowpages.com is a large company with a great many resources at its disposal. As a startup, we can't just throw money at the problem, since we don't have any.<BR/><BR/>I'll keep you updated on our experience with Rails. I'm hopeful, just being reserved with our approach.Leftyhttp://www.blogger.com/profile/14637607058852159739noreply@blogger.comtag:blogger.com,1999:blog-288858287590827405.post-30798811036772014632008-12-24T18:26:00.000-08:002008-12-24T18:26:00.000-08:00Independent of which technology you choose to build your application in, sooner or later you will run in scalability issues. Rails is no different when it comes to scalability than anything else out there. Based on what I read here, you haven't done any performance optimizations yet. The fastest and easiest way to improve performance is to add caching – hitting database with every request will be slow no matter how you look at it. Before going to memcache-like solutions, I suggest looking at Rails built-in caching support first, which could be added without major changes (http://guides.rubyonrails.org/caching_with_rails.html). Second, I am guessing, you haven't optimizations your database queries and database server either – you can significantly speed up MySQL with proper configuration and indexes.<BR/>Finally, if you still think Rails might not scale, instead of looking at twitter.com look at yellowpages.com – they get about 40M unique visitors a month (~10x what twitter gets) and, you guessed right, they do run Rails.Genehttp://www.blogger.com/profile/05443943086571159698noreply@blogger.com