Compression Technique on Crawled Data for High Performance of Web Spider

Web spider is one of the most important components of the search engine, which has been used for accessing the web pages through the World Wide Web(WWW). When we crawl the web pages with different data sizes, then we can access limited web pages according to transfer rate of total data size in per unit time. In case of increasing the data size of web pages, number of web pagesdecrease. So it affects the performance of web spider. This paper describes the issue of data size of web pages, crawled by the web spider. This problem has been implemented here regarding to achieve high performance of web spider by using compression technique, which reduces the data size with big compression ratio of crawled web pages.