Performance issues with using all front-end Web servers for crawling
Crawling content in your farm places a heavy load on the front-end Web servers. This tends to cause spikes and surges in network traffic, and is CPU and memory intensive. Crawling content in your farm can potentially cause more traffic on the network than user requests. This network traffic can negatively impact the performance of all front-end Web servers in your server farm, and thereby impact end-user's requests for content from your SharePoint sites.
Recommended solution
We recommend that you use a dedicated front-end Web server for crawling, especially if you are crawling a server farm that contains more than 500 gigabytes (GB) of content.
Note:
You can specify any front-end Web server in your farm that is not part of a network load balanced cluster as a dedicated front-end Web server for crawling.
For best performance, we recommend that you configure the index server as the dedicated front-end Web server for crawling if your index server has the memory capacity for both roles. By using the same server as both the index server and dedicated front-end Web server, you eliminate the need for the index server to send requests to a different server when crawling content. Consequently, you boost crawl performance and reduce the overall traffic on the network. If this is not possible, you can use a different server in your server farm.
Note:
You cannot configure earlier versions of SharePoint Products and Technologies to use a dedicated front-end Web server and index server on the same computer.
We recommend that you do not include the dedicated front-end Web server in the network load balancing rotation for incoming user requests. Otherwise, user requests that the network load balancer directs to the dedicated front-end Web server for crawling will be subjected to inconsistent performance, as described earlier in this article.
When not to configure a dedicated front-end Web server for crawling
Do not configure a dedicated front-end Web server for crawling if the any of the following conditions exist:
· Another application is running on the index server. Doing so might mean that those applications (such as the Excel Calculation service) can no longer communicate with other servers in the farm.
If other applications are running on the index server, you must first move those applications to another application server before configuring a dedicated front-end Web server for crawling.· The index server is not also the query server. Configuring a dedicated front-end Web server for crawling can disable the index server's ability to propagate the index to another server.