Patent for triggering web crawling based on registry data — Verisign gets
.Com registry awarded patent for using registry data to trigger web crawling.
With Verisign analyzing how to profit from its patent portfolio, competitors ought to pay close attention to the firm’s fresh patents.
Today Verisign was awarded U.S. Patent 8,433,700 (pdf) for “Method and system for triggering web crawling based on registry data”.
The patent describes ways in which a registry can provide data to other firms, such as search engines, that may help them determine when to crawl a domain name’s content.
For example, a registry can provide information regarding fresh domain registrations, registrar transfers, and deletions that can be used by search engines to initiate crawling. A domain transfer may indicate a change in ownership and a site’s use, and thus a need to recrawl the domain.
The service could also use resolution request data to indicate active web sites that ought to be crawled. In one example explained in the patent, increased traffic to a blog could also suggest fresh comments are being left on a blog post. This could warrant speedier recrawling.
In Verisign’s case with .com, it operates a thin registry so it’s zone file doesn’t include much information regarding each domain. The patent allows that additional data resources could be included to provide even more triggers:
Other data sources include data received from registrars such as information that registrant has changed hands, that a registrant has changed hosting firms, data received from hosting firms related to changes to the hosted websites such as file uploading data. These data feeds may be free and publicly available or may be purchased and then integrated into the systems described herein. Therefore, the quality of the data used to trigger crawls may be improved by supplementing the data discussed herein, e.g., changes to the registry database, with additional data useful to the entity performing the crawling.
