Help with EstiBot bulk check php script problem

estibot.com · 12-02-2007, 05:08 AM

Hi,

I want to improve the EstiBot bulk check feature, to increase the number of domains you can check at a time to 1,000.

However, I've encountered a weird problem.

I have a working prototype script (unfortunately I can't publish the code here as it's my competitive edge), but it always terminates after line 693. That is to say, it will evaluate 693 domains, then stop execution of the script. The rest of the script output, which would show various options for the user, is not displayed at all.

It uses PHP and mySQL. Is this some server side problem due to "overheating" by the mySQL queries or something? Any ideas?

Whoever can help me to solve this will get a free link from EstiBot.com, plus 500 namebucks&rep (if incentives are allowed here)
????: NamePros.com http://www.namepros.com/programming/402272-resolved-help-estibot-bulk-check-php.html

Thanks much
Josh

mvl · 12-02-2007, 05:45 AM

I can think of two things: you are either running out of resources or some maximum execution time has passed. But to be honest the cause of your problem could be a million things and I'd have to look deeper into your code, and maybe in the serverlogs to find what it is.

If yoy have more info I'll be back online within a few hours.

-NC- · 12-02-2007, 05:47 AM

always 693?

wild stab in the dark one: you're hitting a memory limit, make it bigger or even unlimited!
ini_set('memory_limit', -1);

wild stab in the dark two: you're hitting an execution time limit, make it bigger
ini_set("max-execution-time" , 60000);

estibot.com · 12-02-2007, 05:47 AM

Thanks Leonardo, I'll look into the max execution time especially. I'll let you know what I find. Resources should not be a problem (at least in theory), as I have a solid dedicated server practically for EstiBot alone.

edit: Thanks-NC- I'll look at both of those

yes, funnily enough always 693, only takes about 20 secs.

Josh

OK, I changed memory limit to -1 and execution time to 6000 secs in php.ini, but still the same problem.

It looks like it's not always exactly 693. Just now I had 692 and 694. Must be an execution time issue? But it takes only 15-20 secs and I have the limit set at 6000 secs now....

Weird.

Peter · 12-02-2007, 06:22 AM

Just to understand a bit better.

The domain list are they in the database at this point or are you reading from user input?

If you are reading from user input what is the method used (presumably post or files as I think get would fail a lot earlier).

If it is post ensure you are not reaching the maximum memory allowed for post (this can be found in post_max_size setting in php.ini).

If you are retrieving from the database at the point of the problem does the script prematurely stop as in not completing the rest of the code or does it complete the rest of the code as if it had finished with the sql result.

jido · 12-02-2007, 06:56 AM

Originally Posted by peter@flexiwebhost

If it is post ensure you are not reaching the maximum memory allowed for post (this can be found in post_max_size setting in php.ini).

To test that read the input from a file instead and see if you get all the lines.

TwistMyArm · 12-02-2007, 07:45 AM

Could it not also be a 'limit' imposed by whatever service you're using to do your lookups?

I mean, if I wrote a whois service and I was getting hit with a flood of whois requests that quickly, I know that I'd block the IP for a while...

estibot.com · 12-02-2007, 08:05 AM

Originally Posted by jido

Originally Posted by peter@flexiwebhost

If it is post ensure you are not reaching the maximum memory allowed for post (this can be found in post_max_size setting in php.ini).

To test that read the input from a file instead and see if you get all the lines.

OK, I will try that.
????: NamePros.com http://www.namepros.com/showthread.php?t=402272

I have made a debugger version that has revealed that the termination does not always occur at the same line in the script. It sometimes stops at the last flush(); and sometimes elsewhere -it doesn't seem to be related to the mySQL call.

Originally Posted by TwistMyArm

Could it not also be a 'limit' imposed by whatever service you're using to do your lookups?

I mean, if I wrote a whois service and I was getting hit with a flood of whois requests that quickly, I know that I'd block the IP for a while...

In this case, no. There are no such limits with this one. Anyway I've had that problem before, and the script didn't terminate, just returned blank results...

Thanks guys, still working.....any other ideas?

Josh

estibot.com · 12-02-2007, 01:16 PM

post max size setting in php.ini is 8M....

Weird. I'm getting 680-695 lines, that's all. I also trimmed a debugging script down to minimal computing intensivity....same problem.

The output says it recognizes 1,003 domains but the script will terminate at around <700

jido · 12-02-2007, 01:33 PM

You have a dedicated server you said? Any relevant message in the web server or system logs?

estibot.com · 12-02-2007, 01:35 PM

I wrote a script that using a for loop displays all the domains that are input by the POST method by the user:

Here is the last few lines of the output:
Piece 998 = chequerun.com
Piece 999 = cherchefoxx.com
Piece 1000 = usedcars.com
Piece 1001 = sex.com

This means that the array is there, the post is OK

However, the appraisal routine itself still terminates at line 680 or so. Must be an MySQL issue...

mvl · 12-02-2007, 01:58 PM

I cannot look into your code but I guess the part where it crashes is structured something like this:

Code:

foreach($domains as $domain) {
       ...
       doSomething();
       ...
       doSomethingWithMySQL();
       ...
       doSomethingWithoutMySQL();
       ...
}

I don't know what you already did but you could try commenting out parts of the code and see what happens. Try to comment out the code that deals with MySQL. If you need results from MySQL substitute them by bogus results.

My approach would be to comment out as much as possible, check if the scripts works correctly and then systematically uncomment code until something gets broken. This way you can try to pinpoint where it goes wrong.

Peter · 12-02-2007, 02:05 PM

what leonardo says is probably going to be your best bet. Without seeing code I can't see how anyone is going to help any further. The main culprits have already been dismissed.

estibot.com · 12-02-2007, 02:41 PM

OK guys. Thanks. I found the culprit. It's PSPELL. When I comment out the pspell routine, the whole thing runs smoothly.

I use pspell for parsing the keywords. It's not the best way, but I don't have any other way presently. It looks like pspell is putting the limit on how many domains can be run in consequence. Any idea how to overcome this?

If you can help with pspell, or suggest some other parsing system, i'd be grateful.

In any case, let's see those links and I'll set them up on the friends page!

Thanks again.

mvl · 12-02-2007, 02:51 PM

Could it be that your dictionaries are corrupt? Does it always crash on the same string?
BTW I don't have a suitable link for you at this moment but I will create something later this week.

jido · 12-05-2007, 06:19 AM

What do you mean "putting a limit on how many domains can be run in consequence"? Could you tell us about your OS (Linux?) and the pspell routines you use so we can try and reproduce.

Do you use preg_replace_callback as suggested here?

I am not sure I helped at all but if you feel like adding my site use the TimeAfter forum link in my sig.

12-02-2007, 05:08 AM	THREAD STARTER #1 (permalink)
estibot.com Developer of Estibot Join Date: Jun 2006 Posts: 1,945	Help with EstiBot bulk check php script problem Hi, I want to improve the EstiBot bulk check feature, to increase the number of domains you can check at a time to 1,000. However, I've encountered a weird problem. I have a working prototype script (unfortunately I can't publish the code here as it's my competitive edge), but it always terminates after line 693. That is to say, it will evaluate 693 domains, then stop execution of the script. The rest of the script output, which would show various options for the user, is not displayed at all. It uses PHP and mySQL. Is this some server side problem due to "overheating" by the mySQL queries or something? Any ideas? Whoever can help me to solve this will get a free link from EstiBot.com, plus 500 namebucks&rep (if incentives are allowed here) ????: NamePros.com http://www.namepros.com/programming/402272-resolved-help-estibot-bulk-check-php.html Thanks much Josh __________________ FREE 40+ page eBook on Domain Development Download now! Start Your Own Domain Registration Business! Estibot Reseller Plans

12-02-2007, 05:45 AM	#2 (permalink)
mvl fka: leonardo Join Date: Aug 2006 Posts: 736	I can think of two things: you are either running out of resources or some maximum execution time has passed. But to be honest the cause of your problem could be a million things and I'd have to look deeper into your code, and maybe in the serverlogs to find what it is. If yoy have more info I'll be back online within a few hours. __________________ http://sxt.nl/ ForeignPropertyForSale.com SATAN.EU VOI.EU http://HealthyDishes.com

12-02-2007, 05:47 AM	#3 (permalink)
-NC- Traveller Join Date: Mar 2007 Location: Yet another city Posts: 1,419	always 693? wild stab in the dark one: you're hitting a memory limit, make it bigger or even unlimited! ini_set('memory_limit', -1); wild stab in the dark two: you're hitting an execution time limit, make it bigger ini_set("max-execution-time" , 60000); __________________ NameCooler.com

12-02-2007, 05:47 AM	THREAD STARTER #4 (permalink)
estibot.com Developer of Estibot Join Date: Jun 2006 Posts: 1,945	Thanks Leonardo, I'll look into the max execution time especially. I'll let you know what I find. Resources should not be a problem (at least in theory), as I have a solid dedicated server practically for EstiBot alone. edit: Thanks-NC- I'll look at both of those yes, funnily enough always 693, only takes about 20 secs. Josh OK, I changed memory limit to -1 and execution time to 6000 secs in php.ini, but still the same problem. It looks like it's not always exactly 693. Just now I had 692 and 694. Must be an execution time issue? But it takes only 15-20 secs and I have the limit set at 6000 secs now.... Weird. __________________ FREE 40+ page eBook on Domain Development Download now! Start Your Own Domain Registration Business! Estibot Reseller Plans

12-02-2007, 06:22 AM	#5 (permalink)
Peter NamePros Expert Join Date: Nov 2003 Location: Scotland Posts: 5,069	Just to understand a bit better. The domain list are they in the database at this point or are you reading from user input? If you are reading from user input what is the method used (presumably post or files as I think get would fail a lot earlier). If it is post ensure you are not reaching the maximum memory allowed for post (this can be found in post_max_size setting in php.ini). If you are retrieving from the database at the point of the problem does the script prematurely stop as in not completing the rest of the code or does it complete the rest of the code as if it had finished with the sql result. __________________ Manage your portfolio using my new Domain Portfolio Management script. Securing Your Domain Name From Theft

12-02-2007, 07:45 AM	#7 (permalink)
TwistMyArm NamePros Member Join Date: May 2006 Posts: 160	Could it not also be a 'limit' imposed by whatever service you're using to do your lookups? I mean, if I wrote a whois service and I was getting hit with a flood of whois requests that quickly, I know that I'd block the IP for a while... __________________ Free Fuel Efficiency Calculator \| Buy CMS Templates

12-02-2007, 01:16 PM	THREAD STARTER #9 (permalink)
estibot.com Developer of Estibot Join Date: Jun 2006 Posts: 1,945	post max size setting in php.ini is 8M.... Weird. I'm getting 680-695 lines, that's all. I also trimmed a debugging script down to minimal computing intensivity....same problem. The output says it recognizes 1,003 domains but the script will terminate at around <700 __________________ FREE 40+ page eBook on Domain Development Download now! Start Your Own Domain Registration Business! Estibot Reseller Plans

12-02-2007, 01:33 PM	#10 (permalink)
jido Senior Member Join Date: Aug 2007 Posts: 2,172	You have a dedicated server you said? Any relevant message in the web server or system logs? __________________ ______________________________________ Help with crosswords http://lettercombination.com Pitfalls of Java http://javasyntax.com _______________ f o r . s a l e ______________

12-02-2007, 01:35 PM	THREAD STARTER #11 (permalink)
estibot.com Developer of Estibot Join Date: Jun 2006 Posts: 1,945	I wrote a script that using a for loop displays all the domains that are input by the POST method by the user: Here is the last few lines of the output: Piece 998 = chequerun.com Piece 999 = cherchefoxx.com Piece 1000 = usedcars.com Piece 1001 = sex.com This means that the array is there, the post is OK However, the appraisal routine itself still terminates at line 680 or so. Must be an MySQL issue... __________________ FREE 40+ page eBook on Domain Development Download now! Start Your Own Domain Registration Business! Estibot Reseller Plans

12-02-2007, 01:58 PM	#12 (permalink)
mvl fka: leonardo Join Date: Aug 2006 Posts: 736	I cannot look into your code but I guess the part where it crashes is structured something like this: Code: foreach($domains as $domain) { ... doSomething(); ... doSomethingWithMySQL(); ... doSomethingWithoutMySQL(); ... } I don't know what you already did but you could try commenting out parts of the code and see what happens. Try to comment out the code that deals with MySQL. If you need results from MySQL substitute them by bogus results. My approach would be to comment out as much as possible, check if the scripts works correctly and then systematically uncomment code until something gets broken. This way you can try to pinpoint where it goes wrong. __________________ http://sxt.nl/ ForeignPropertyForSale.com SATAN.EU VOI.EU http://HealthyDishes.com

12-02-2007, 02:05 PM	#13 (permalink)
Peter NamePros Expert Join Date: Nov 2003 Location: Scotland Posts: 5,069	what leonardo says is probably going to be your best bet. Without seeing code I can't see how anyone is going to help any further. The main culprits have already been dismissed. __________________ Manage your portfolio using my new Domain Portfolio Management script. Securing Your Domain Name From Theft

12-02-2007, 02:41 PM	THREAD STARTER #14 (permalink)
estibot.com Developer of Estibot Join Date: Jun 2006 Posts: 1,945	OK guys. Thanks. I found the culprit. It's PSPELL. When I comment out the pspell routine, the whole thing runs smoothly. I use pspell for parsing the keywords. It's not the best way, but I don't have any other way presently. It looks like pspell is putting the limit on how many domains can be run in consequence. Any idea how to overcome this? If you can help with pspell, or suggest some other parsing system, i'd be grateful. In any case, let's see those links and I'll set them up on the friends page! Thanks again. __________________ FREE 40+ page eBook on Domain Development Download now! Start Your Own Domain Registration Business! Estibot Reseller Plans

12-02-2007, 02:51 PM	#15 (permalink)
mvl fka: leonardo Join Date: Aug 2006 Posts: 736	Could it be that your dictionaries are corrupt? Does it always crash on the same string? BTW I don't have a suitable link for you at this moment but I will create something later this week. __________________ http://sxt.nl/ ForeignPropertyForSale.com SATAN.EU VOI.EU http://HealthyDishes.com

12-05-2007, 06:19 AM	#16 (permalink)
jido Senior Member Join Date: Aug 2007 Posts: 2,172	What do you mean "putting a limit on how many domains can be run in consequence"? Could you tell us about your OS (Linux?) and the pspell routines you use so we can try and reproduce. Do you use preg_replace_callback as suggested here? I am not sure I helped at all but if you feel like adding my site use the TimeAfter forum link in my sig. __________________ ______________________________________ Help with crosswords http://lettercombination.com Pitfalls of Java http://javasyntax.com _______________ f o r . s a l e ______________

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)