[Resolved] Help with EstiBot bulk check Php script problem

[estibot.com]

Help with EstiBot bulk check php script problem

Hi,

I want to improve the EstiBot bulk check feature, to increase the number of domains you can check at a time to 1,000.

However, I've encountered a weird problem.

I have a working prototype script (unfortunately I can't publish the code here as it's my competitive edge), but it always terminates after line 693. That is to say, it will evaluate 693 domains, then stop execution of the script. The rest of the script output, which would show various options for the user, is not displayed at all.

It uses PHP and mySQL. Is this some server side problem due to "overheating" by the mySQL queries or something? Any ideas?

Whoever can help me to solve this will get a free link from EstiBot.com, plus 500 namebucks&rep (if incentives are allowed here)

Thanks much
Josh

 

[mvl]

I can think of two things: you are either running out of resources or some maximum execution time has passed. But to be honest the cause of your problem could be a million things and I'd have to look deeper into your code, and maybe in the serverlogs to find what it is.

If yoy have more info I'll be back online within a few hours.

 

[-NC-]

always 693?

wild stab in the dark one: you're hitting a memory limit, make it bigger or even unlimited!
ini_set('memory_limit', -1);

wild stab in the dark two: you're hitting an execution time limit, make it bigger
ini_set("max-execution-time" , 60000);

 

[estibot.com]

Thanks Leonardo, I'll look into the max execution time especially. I'll let you know what I find. Resources should not be a problem (at least in theory), as I have a solid dedicated server practically for EstiBot alone.

edit: Thanks-NC- I'll look at both of those

yes, funnily enough always 693, only takes about 20 secs.

Josh

OK, I changed memory limit to -1 and execution time to 6000 secs in php.ini, but still the same problem.

It looks like it's not always exactly 693. Just now I had 692 and 694. Must be an execution time issue? But it takes only 15-20 secs and I have the limit set at 6000 secs now....

Weird.

 

[Peter]

Just to understand a bit better.

The domain list are they in the database at this point or are you reading from user input?

If you are reading from user input what is the method used (presumably post or files as I think get would fail a lot earlier).

If it is post ensure you are not reaching the maximum memory allowed for post (this can be found in post_max_size setting in php.ini).

If you are retrieving from the database at the point of the problem does the script prematurely stop as in not completing the rest of the code or does it complete the rest of the code as if it had finished with the sql result.

 

[jido]

If it is post ensure you are not reaching the maximum memory allowed for post (this can be found in post_max_size setting in php.ini).

To test that read the input from a file instead and see if you get all the lines.

 

[TwistMyArm]

Could it not also be a 'limit' imposed by whatever service you're using to do your lookups?

I mean, if I wrote a whois service and I was getting hit with a flood of whois requests that quickly, I know that I'd block the IP for a while...

 

[estibot.com]

If it is post ensure you are not reaching the maximum memory allowed for post (this can be found in post_max_size setting in php.ini).

To test that read the input from a file instead and see if you get all the lines.

OK, I will try that.

I have made a debugger version that has revealed that the termination does not always occur at the same line in the script. It sometimes stops at the last flush(); and sometimes elsewhere -it doesn't seem to be related to the mySQL call.

Could it not also be a 'limit' imposed by whatever service you're using to do your lookups?

I mean, if I wrote a whois service and I was getting hit with a flood of whois requests that quickly, I know that I'd block the IP for a while...

In this case, no. There are no such limits with this one. Anyway I've had that problem before, and the script didn't terminate, just returned blank results...

Thanks guys, still working.....any other ideas?

Josh

 

[estibot.com]

post max size setting in php.ini is 8M....

Weird. I'm getting 680-695 lines, that's all. I also trimmed a debugging script down to minimal computing intensivity....same problem.

The output says it recognizes 1,003 domains but the script will terminate at around <700

 

[jido]

You have a dedicated server you said? Any relevant message in the web server or system logs?

 

[estibot.com]

I wrote a script that using a for loop displays all the domains that are input by the POST method by the user:

Here is the last few lines of the output:
Piece 998 = chequerun.com
Piece 999 = cherchefoxx.com
Piece 1000 = usedcars.com
Piece 1001 = sex.com

This means that the array is there, the post is OK

However, the appraisal routine itself still terminates at line 680 or so. Must be an MySQL issue...

 

[mvl]

I cannot look into your code but I guess the part where it crashes is structured something like this:

foreach($domains as $domain) {
       ...
       doSomething();
       ...
       doSomethingWithMySQL();
       ...
       doSomethingWithoutMySQL();
       ...
}

I don't know what you already did but you could try commenting out parts of the code and see what happens. Try to comment out the code that deals with MySQL. If you need results from MySQL substitute them by bogus results.

My approach would be to comment out as much as possible, check if the scripts works correctly and then systematically uncomment code until something gets broken. This way you can try to pinpoint where it goes wrong.

 

[Peter]

what leonardo says is probably going to be your best bet. Without seeing code I can't see how anyone is going to help any further. The main culprits have already been dismissed.

 

[estibot.com]

OK guys. Thanks. I found the culprit. It's PSPELL. When I comment out the pspell routine, the whole thing runs smoothly.

I use pspell for parsing the keywords. It's not the best way, but I don't have any other way presently. It looks like pspell is putting the limit on how many domains can be run in consequence. Any idea how to overcome this?

If you can help with pspell, or suggest some other parsing system, i'd be grateful.

In any case, let's see those links and I'll set them up on the friends page!

Thanks again.

 

[mvl]

Could it be that your dictionaries are corrupt? Does it always crash on the same string?
BTW I don't have a suitable link for you at this moment but I will create something later this week.

 

[jido]

What do you mean "putting a limit on how many domains can be run in consequence"? Could you tell us about your OS (Linux?) and the pspell routines you use so we can try and reproduce.

Do you use preg_replace_callback as suggested here?

I am not sure I helped at all but if you feel like adding my site use the TimeAfter forum link in my sig.