Changing IP When Parsing in PHP
Due to the delays inserted during parsing, the speed of the parser is significantly reduced. It can, however, be increased if you have several IP addresses. Then each request to the site can be made from a different IP, which will allow you to set a shorter delay time (by a factor of how many IP addresses you have). There are two ways to do this.
The first way is to buy additional IP addresses from your hosting provider. In this case, switching IPs during parsing is done with the following CURL option:
<?php
curl_setopt($curl, CURLOPT_INTERFACE, 'ip address');
?>
The second way is to buy proxy servers from special services (you can use free proxies, but as practice shows - they mostly don't work). In the case of proxies, switching the ip during parsing is done with the following CURL option:
<?php
curl_setopt($curl, CURLOPT_PROXY, 'ip address');
?>
Write a parser that will work with changing IP addresses.
The target site may ban one or more of your addresses during parsing. Add a check that will verify if the IP is working and temporarily remove it from the list if it is banned.
The more the target site bans your IPs, the more the load on the remaining ones increases. Make it so that the delay time increases as the number of IP addresses decreases.
