Answers
Oct 03, 2006 - 01:40 PM
PHP can scale, many say that it cannot, however i have seen it scale on very large websites. Many say that Coldfusion doesnt scale well, but myspace managed it.
Try to hardcode as little as possiable, makre sure things like database username passwords hosts etc can be changed easily from one file (outside the document root if possiable). Remember to test your scripts for speed, look at different ays of doing things to see if there is a faster way. A common one that people do is the following
$sql="select * from blah";
$query=mysql_query($sql);
$count=mysql_num_rows($query);
Now when you have a small site, that will seem fairly fast, and it wont worry you, however on a large site you will find the following much faster
$sql="select count(field) from blah";
$query=mysql_query($sql);
list($count) = mysql_fetch_row($query);
Things like this make very little difference on small sites, but as sites grow they make a huge difference.
It is also worth looking into memcached, as it can speed up PHP/MySQL applications compared to accessing the database directly.
Keep your code clean, keep plenty of functions and classes where they make sense, keep config seperate from the code, keep the layout seperate from the code, and you should be able to scale using clusters at a later stage with little effort.
Try to hardcode as little as possiable, makre sure things like database username passwords hosts etc can be changed easily from one file (outside the document root if possiable). Remember to test your scripts for speed, look at different ays of doing things to see if there is a faster way. A common one that people do is the following
$sql="select * from blah";
$query=mysql_query($sql);
$count=mysql_num_rows($query);
Now when you have a small site, that will seem fairly fast, and it wont worry you, however on a large site you will find the following much faster
$sql="select count(field) from blah";
$query=mysql_query($sql);
list($count) = mysql_fetch_row($query);
Things like this make very little difference on small sites, but as sites grow they make a huge difference.
It is also worth looking into memcached, as it can speed up PHP/MySQL applications compared to accessing the database directly.
Keep your code clean, keep plenty of functions and classes where they make sense, keep config seperate from the code, keep the layout seperate from the code, and you should be able to scale using clusters at a later stage with little effort.
Oct 03, 2006 - 02:21 PM
Sorry a few other things that i want to add.
Asking what a single server can handle is pretty hard to explain, as it depends on the spec. If you had a little P3 then it would be nothing compared to a sun primepower. It would also depend what you where running, MySQL uses a lot of resources, but apache not so much.
If you reach the limit of one server and dont have a coupleof million spare for a sun server (after all the extras) then you have to cluster it. If everything is stored in mysql databases then this is easy, if you have people uploading images then you will be best storing the full URL to images (say server1.domain.com/img1.ext) then when you cluster you can start storing them on server2, server3 etc etc)
Asking what a single server can handle is pretty hard to explain, as it depends on the spec. If you had a little P3 then it would be nothing compared to a sun primepower. It would also depend what you where running, MySQL uses a lot of resources, but apache not so much.
If you reach the limit of one server and dont have a coupleof million spare for a sun server (after all the extras) then you have to cluster it. If everything is stored in mysql databases then this is easy, if you have people uploading images then you will be best storing the full URL to images (say server1.domain.com/img1.ext) then when you cluster you can start storing them on server2, server3 etc etc)
Oct 03, 2006 - 05:24 PM
Very interesting mr. little,
Regarding your first response, I do know how to code php and sql, and besides I guess the issues you mention apply to the alternative languages as well.
What I do know nothing about is clustering, so I really appreciate your advice here - what to do and not to do to make that possible. Can you explain more about that?
I guess it means that when a user connects to my site, he will end up being served by one or another physical computer, however he will not notice the difference - is it something I should be taking into consideration when developing?
Another thing - the MySQL database. What is the limit of rows in a table and is there a limit of storage space? Is the MySQL server and data also easily clusterable?
Regarding your first response, I do know how to code php and sql, and besides I guess the issues you mention apply to the alternative languages as well.
What I do know nothing about is clustering, so I really appreciate your advice here - what to do and not to do to make that possible. Can you explain more about that?
I guess it means that when a user connects to my site, he will end up being served by one or another physical computer, however he will not notice the difference - is it something I should be taking into consideration when developing?
Another thing - the MySQL database. What is the limit of rows in a table and is there a limit of storage space? Is the MySQL server and data also easily clusterable?
Oct 04, 2006 - 05:48 AM
I wasnt trying to insult your intelegence, oviously i dont know how much experence you have had. The last person i talked to about this had just finished doing hello world, so i hope your didn't take offence.
Clustering is not too hard, but it depends on the application, you can acutlaly setup DNS load balancing your self, which is a good way to test your application in advanced without needing dedicated servers, you can just use shared hosting accounts.
Create 2 A records for your domain www1 and www2 and point them to the correct IP addresses, then instead of an A record for www create 2 CNAME records pointing to www1 and www2, and there you have it a DNS round robin load balaced site.
From the coding point of view, If users are not uploading things, its very easy, just a simple case of coping the code to another server, and getting it to use the same mysql database.
If they are uploading things, you really have 2 choices, 1 get a good storange server on the backend that all servers connect to, or 2 do the balancing yourself, to do that you would need to monitor the disk usage, and get your script to change what server you upload images to. For example, when WWW1 is getting full, you make a small change in your code so that instead of uploads posting directly to WWW1 (this will bypass the load balancing, so that its forced to a perticular server) you get it to post directly to WWW2. doing this means that you must store the server the image is on in the database. Of course you dont actually have to use filebased storage, you could store images as raw data in the database, then has a PHP script to pull it from the database (so you have say image.php?imagename.jpg) this would also work, as when you use mysql cluster it would be avaliable on all servers all the time.
Mysql clusters very well, however if your not a server admin, you may wish to grab some help. If you think you may use mysql then you can use the same idea with your DNS as used for the www record. So you create db1 and db2 as A records then have multi CNAMES for db. Then you get php to connect to db.domain.com as the host instead of an IP address.
I have personally run a mysql database with 2 million records in a large table, it mysql was not to bothered about it, the only real limitation is the size of the HDD, which with 500GB HDD's avaliable and with RAID, you can get a pretty huge amount of storage.
The main thing you need to consider is how you are going to do your loadbalancing. DNS load balancing works, but if a server goes down, you will still have and outage. This is where hardware loadblancing and clustering comes into play, but you can still use the DNS to make upgrading easy.
If you consider that you are already too big for a single server, then we can simulate what you would need.
Personally i would recomend 2 webservers, 2 database servers and 1 master server.
The master server handles your email, ftp and also runs a few programs to make the clustering work (such as the mysql cluster manager program (cannot remember the exact name just now).
When you upload a file by FTP, you would need to setup a script to replicate that file onto both webservers which is not too complex. This would keep your scripts upto date, for client uploads, i would recomend using mysql storage as then you do not have to deal with moving client uploads between servers, or force them to upload to one server and store which server they are on. Your management server would also run a program to monitor the mysql cluster so that if a node goes down, it can be restored.
When developing make sure you use PHP5 and the latest mysql you can (4.1 i think it around a lot) as this will make clustering easier (If you use mysql3 you will have problems with your SQL syntax when you move to mysql cluster).
Hope this helps
Clustering is not too hard, but it depends on the application, you can acutlaly setup DNS load balancing your self, which is a good way to test your application in advanced without needing dedicated servers, you can just use shared hosting accounts.
Create 2 A records for your domain www1 and www2 and point them to the correct IP addresses, then instead of an A record for www create 2 CNAME records pointing to www1 and www2, and there you have it a DNS round robin load balaced site.
From the coding point of view, If users are not uploading things, its very easy, just a simple case of coping the code to another server, and getting it to use the same mysql database.
If they are uploading things, you really have 2 choices, 1 get a good storange server on the backend that all servers connect to, or 2 do the balancing yourself, to do that you would need to monitor the disk usage, and get your script to change what server you upload images to. For example, when WWW1 is getting full, you make a small change in your code so that instead of uploads posting directly to WWW1 (this will bypass the load balancing, so that its forced to a perticular server) you get it to post directly to WWW2. doing this means that you must store the server the image is on in the database. Of course you dont actually have to use filebased storage, you could store images as raw data in the database, then has a PHP script to pull it from the database (so you have say image.php?imagename.jpg) this would also work, as when you use mysql cluster it would be avaliable on all servers all the time.
Mysql clusters very well, however if your not a server admin, you may wish to grab some help. If you think you may use mysql then you can use the same idea with your DNS as used for the www record. So you create db1 and db2 as A records then have multi CNAMES for db. Then you get php to connect to db.domain.com as the host instead of an IP address.
I have personally run a mysql database with 2 million records in a large table, it mysql was not to bothered about it, the only real limitation is the size of the HDD, which with 500GB HDD's avaliable and with RAID, you can get a pretty huge amount of storage.
The main thing you need to consider is how you are going to do your loadbalancing. DNS load balancing works, but if a server goes down, you will still have and outage. This is where hardware loadblancing and clustering comes into play, but you can still use the DNS to make upgrading easy.
If you consider that you are already too big for a single server, then we can simulate what you would need.
Personally i would recomend 2 webservers, 2 database servers and 1 master server.
The master server handles your email, ftp and also runs a few programs to make the clustering work (such as the mysql cluster manager program (cannot remember the exact name just now).
When you upload a file by FTP, you would need to setup a script to replicate that file onto both webservers which is not too complex. This would keep your scripts upto date, for client uploads, i would recomend using mysql storage as then you do not have to deal with moving client uploads between servers, or force them to upload to one server and store which server they are on. Your management server would also run a program to monitor the mysql cluster so that if a node goes down, it can be restored.
When developing make sure you use PHP5 and the latest mysql you can (4.1 i think it around a lot) as this will make clustering easier (If you use mysql3 you will have problems with your SQL syntax when you move to mysql cluster).
Hope this helps
Oct 04, 2006 - 11:05 AM
Very helpful indeed.
No offence taken, and I hope you excuse me for my direct way of getting to my point :-)
I am also not pretending to be an expert coder - I have many things to learn yet!
The fundamental question though: IN PRINCIPLE, if php and mysql are as easily clustered as you claim, it doesn't matter if you don't code very optimized sql's as long as you just have the economy, hardware and knowledge to enforce the serverpower?
I will be getting right back to you with more little comments to your fine explanations.
Thank,
Jakob
No offence taken, and I hope you excuse me for my direct way of getting to my point :-)
I am also not pretending to be an expert coder - I have many things to learn yet!
The fundamental question though: IN PRINCIPLE, if php and mysql are as easily clustered as you claim, it doesn't matter if you don't code very optimized sql's as long as you just have the economy, hardware and knowledge to enforce the serverpower?
I will be getting right back to you with more little comments to your fine explanations.
Thank,
Jakob
Oct 04, 2006 - 11:11 AM
oh, if you have money to throw away, then no it does not matter if your code and sqls are optimized.
However, when you run a business, you need to think of costs. If you build a big site that is optimized, you may need 1 server. If none of its optimized then you might need 10 server (not very accurate as it would depend on the size of the system you built).
Clustering is not easy, unless you know how to do it. Setting up the servers is relitively easy, lots of tuorials about that (for mysql cluster and rysncing files between servers for your code updates) its the load balancers that are not always easy.
Always plan ahead, try to think if this code was to run on 2 seperate servers, would it work. One problem for example is sessions may not work, so you may have to set cookies and manage your own sessions in the database, rather than PHPs sessions.
To develop the system, i would highly recomend getting 2 shared accounts on different servers as you will be able to simulate what happens when your code is on 2 different servers that way.
However, when you run a business, you need to think of costs. If you build a big site that is optimized, you may need 1 server. If none of its optimized then you might need 10 server (not very accurate as it would depend on the size of the system you built).
Clustering is not easy, unless you know how to do it. Setting up the servers is relitively easy, lots of tuorials about that (for mysql cluster and rysncing files between servers for your code updates) its the load balancers that are not always easy.
Always plan ahead, try to think if this code was to run on 2 seperate servers, would it work. One problem for example is sessions may not work, so you may have to set cookies and manage your own sessions in the database, rather than PHPs sessions.
To develop the system, i would highly recomend getting 2 shared accounts on different servers as you will be able to simulate what happens when your code is on 2 different servers that way.
Oct 08, 2006 - 08:20 PM
I've tried to store images in the database before - I like the idea of not worrying about files - but wasn't very impressed with the performance. Every time an image is loaded, it takes php and mysql ressources. Browser caching and downloading was also complicated. So this time I intended to l go with the filebased image storing which means that once all the filenames have been retracted from the database, the user can view a whole gallery without bothering anything but the file server. Does that make sense?
I'll have to think about what you say about sessions. I guess the problem is that if I use the DNS load balancing as you describe, the user might end up on different servers for each request. Do you know if there are other ways of clustering, where normal server handled sessions are not a problem? I.e. if the users requests is always sent to the same server?
I'll have to think about what you say about sessions. I guess the problem is that if I use the DNS load balancing as you describe, the user might end up on different servers for each request. Do you know if there are other ways of clustering, where normal server handled sessions are not a problem? I.e. if the users requests is always sent to the same server?
Oct 22, 2006 - 03:31 PM
If you have the budjet, it might be interesting for you to check out something like the zend platform. The fact that it has tools to help you cluster servers and profile the bottlenecks in your applications (mysql queries and php scripts) is impressive.
They have a live demo:
http://www.zend.com/products/zend_pla...
Good luck
They have a live demo:
http://www.zend.com/products/zend_pla...
Good luck
Nov 28, 2013 - 02:52 AM
There are lots of resources on the web that explain how to scale web applications and how to scale Rails. Here's a quick summary of just four of the basic strategies for a scalable web application. For nore info https://www.engineyard.com/cpages/lar...
Dec 20, 2014 - 03:26 AM
Php not support SQL NULL type in large scale web applications and there is No support for the full range of SDO data types.In large scale web application php supports one foreign key per table.
Add New Comment