Solved Data Extraction problem by adding "status" column

Recently I had worked on developing a Script for getting data from external websites using php cURL.

It took more Time and Effort for executing the script, as the script execution was stopped intermittently many time due to unknown issue.

i-e I had to rerun the script multiple times to collect the data from the third-party site.

I couldn't find the reason why the script execution was stopped. Anyway, I found a work around to handle this issue.
I updated the php script to store the extracted data in a mysql table with status column. i-e once data for particular item is extracted fully, the corresponding row in the  mysql table will me marked as "completed". And, updated the php script to process only the rows which are NOT having status as "completed".

This workaround saved lot of time and effort. Anyway, I am still looking for the root cause of this issue.  I already set huge value as script time out period using "set_time_limit()" function.



You can subscribe to our Email posts, and you can bookmark this blog for further reading, or you can subscribe to our blog feed.