Text 4066, 74 rader
Skriven 2005-05-09 19:08:22 av Mike '/m' (1:379/45)
   Kommentar till text 4059 av Tony Ingenoso (1:379/45)
Ärende: Re: Google is cool, Microsoft isn't
===========================================
From: Mike '/m' <mike@barkto.com>


Good article on what google does and how they do it...

http://www.internetnews.com/xSP/article.php/3487041

===
BURLINGAME, Calif. -- The key to the speed and reliability of Google (Quote,
Chart) search is cutting up data into chunks, its top engineer said.

Urs Hoelzle, Google vice president of operations and vice president of
engineering, offered a rare behind-the-scenes tour of Google's architecture on
Wednesday. Hoelzle spoke here at EclipseCon 2005, a conference on the open
source, extensible platform for software tools.

To deal with the more than 10 billion Web pages and tens of terabytes of
information on Google's servers, the company combines cheap machines with
plenty of redundancy, Hoelzle said. Its commodity servers cost around $1,000
apiece, and Google's architecture places them into interconnected nodes.

All machines run on a stripped-down Linux kernel. The distribution is Red Hat
(Quote, Chart), but Hoelzle said Google doesn't use much of the distro.
Moreover, Google has created its own patches for things that haven't been fixed
in the original kernel.

"The downside to cheap machines is, you have to make them work together
reliably," Hoelzle said. "These things are cheap and easy to put together. The
problem is, these things break."

In fact, at Google, many will fail every day. So, Google has automated methods
of dealing with machine failures, allowing it to build a fast, highly reliable
service with cheap hardware.

Google replicates the Web pages it caches by splitting them up into pieces it
calls "shards." The shards are small enough that several can fit on one
machine. And they're replicated on several machines, so that if one breaks,
another can serve up the information. The master index is also split up among
several servers, and that set also is replicated several times. The engineers
call these "chunk servers"....
===

The entire article is well worth the read.

 /m


On Mon, 9 May 2005 01:15:32 -0400, "Tony Ingenoso" <admin@spamcop.net> wrote:

>Given the CPU cycles Google has to burn doing what they do, they ARE keenly
>interested in those specialties and will be actively hacking their platforms
>and tools to suit their application.
>
>I spent a while at IDC (a division of Chase at the time) in Waltham 20 years
>ago.  IDC was running a heavily hacked VM/CMS of their own construction at
>the time tuned to their particular needs.  People do these kind of things
>when they're pushing the existing software platforms to their limits.
>
>Suppose Google could implement some particualr compiler opimizations that
>speed things up measurably?  That allows them to keep existing hardware in
>place a while longer and defer a H/W replacement cycle.  They've got a lot
>of machines.  Replacing/adding to them all the time is expensive.
>
>The Google approach of using lots of cheap crap lends itself well to
>software tweaks - any perf gains are multiplied thousands of  times.
>
>"Mike '/m'" <mike@barkto.com> wrote in message
>news:aeft71db88t00pomqj9n6f3uch1cinpq7v@4ax.com...
>
>>  things like operating-system design, compiler optimization, and
>distributed-systems architecture.

--- BBBS/NT v4.01 Flag-5
 * Origin: Barktopia BBS Site http://HarborWebs.com:8081 (1:379/45)