Tech question(s) about bots?

[Chris2412]

I am a n00b developer and I would like to talk about bots and how they crawl a web page.

I tried searching keyword “bot” on the forum but I got nill, just a bunch of random results. I’m sure this is a thread covering this so a link provided will do just fine.

So, Google has “bots” that crawl your page. It looks for keywords, phrases, ect for cataloging purposes. Which is good, because you want your website cataloged in their search engine.

But there are other kind of bots, too. Yes?

Some of these bots are evil pawns sent out to do- what exactly?

Eat your bandwidth?

I read a simple php code a year or two ago, that basically makes bot’s sleep (or time out).

I am not even close to HTML 5 yet so perhaps I am getting way ahead of myself.

There’s no harm in asking. Perhaps I can bump this thread as I get more knowledge and have additional questions regarding coding.

 

[enlytend]

(NP is a great forum but focuses on a very different area of expertise than what you're asking about - have you tried googling it ?)

Grab email addresses for spam lists
Leave Referrer spam
Scan for hackable vulnerabilities in your site code
Scrape your site content to use it elsewhere
Leave comment spam

There are also "badly behaved" bots which may or may not be a malicious application but
who disobey robots.txt directives. Or crawl your site so frequently and so aggressively that they eat bandwidth and affect performance.

To name a few ...

(BTW, Google, Bing, etc. look at a LOT more than just keywords and phrases.)

 

[iowadawg]

Worse offender, once they find your blog/site?
Baidu!

 

[Chris2412]

enlytend, absolutely I "binged it"... Majority of the results in the search were from persons trying to actually create bots, which is a scary thing.

I tried to post my question in the proper forum.

I do appreciate your response.

If NP forum is not a community to help, advise, and educate in web development...than you are correct- I am in the wrong place.

However, I cannot promise you I won't continue to ask questions. I don't care if I look foolish to you, I am trying to learn. Hopefully for every 1 person that makes fun of me there will be 1 person that can teach me a thing or two. Or three. Or four.

Knowledge is power.

 

[Beezy]

If you use cloudflare on your site they will do a great job of keeping offending bots away (as well as the other benefits).

 

[Chris2412]

Okay, what if we just start from the very beginning like I am a total n00b

What percentage of bots do you want crawling your site?

 

[Beezy]

Okay, what if we just start from the very beginning like I am a total n00b

What percentage of bots do you want crawling your site?

Honestly don't even think about it unless for some reason your host is saying that your database is out of control.

 

[Chris2412]

But if you had to give a number. Just humour me. What % of bot activity is actually "good activity" for a domain?

 

[Beezy]

% is a relative term. What is your traffic/how often do you add new content/etc.

 

[Chris2412]

Okay, fair enough. I don't have those figures because I am not online yet.

Let me ask you this; enlytend specified evil spawn of satan bots hack for "vulnerabilities in your site code".

Can you elaborate on this? What are (common) vulnerabilities?

 

[Beezy]

sql injections, javascript vulnerabilities.

Can you explain where your line of questioning is coming from and maybe we can help you? You aren't making much sense.

 

[Beezy]

Here is a random site that I never update:

Again, not sure what you're getting at so no idea if that's helpful to you.

 

[Chris2412]

Well, I am just trying to understand bots.

Why someone would send them to eat bandwidth?

Why people are engineering bots instead of... basically anything else they could be doing?

Why some bots are "good" and others are evil?

And ultimately... how to code to keep myself safe! Like I said, there was a very simple php code I read before I even started in this field. It basically timed-out the bot. But there are good bots and bad ones, so I don't really know.

 

[Chris2412]

That graph you posted, is that the site you referenced earlier?

 

[Beezy]

Yeah, cloudflare (which is used here on namepros, too) blacklists bad bots.

But even if you don't use cloudflare, like I said above, don't even think about it. You'll do more harm than good using that script unless you aren't following development best practices. Make sure everything is updated and secure and your hosting is good and you'll be fine.

 

[Chris2412]

Thank you, Beezy.

I appreciate your time and your efforts this evening.

Often I feel like there is not enough hours in a day. I am so consumed, like I'm being pulled every which way trying to learn.

Baby steps!

 

[Beezy]

I've been at this for years and I still feel like that every day.

Let me know when you're done building whatever you're working on. I'd love to check it out.

 

[Chris2412]

Deal!

You are not the first person to say that to me on NP.

I love this community!

 

[iowadawg]

All I know is what I read.
And that is that most bots are evil bots.

 

[Chris2412]

All I know is what I read.
And that is that most bots are evil bots.

Okay, so let me ask this- lets say I insert a very simple "script" or php code into my index file making all bots sleep (time-out). Will I lose business? Essentially no bot could crawl me, including the search engines.

I can't see learning to code just to hire a sub-contractor to watchdog my website. That's my general feeling.

 

[Chris2412]

And i do understand what Beezy is saying.

I'm basically asking "what can I do to grow this plant" without giving you any information of geographical or any other variables. Even if I did, would you advise me for free?

So I do understand that, I guess you could say I am asking hypothetically because I don't have specifics for you.

I do appreciate everyone's input, even if you want to razzle with me, it's all fine and good.

 

[Asfas1000]

I think you're overthinking the whole matter, mate.

 

[Chris2412]

I think you're overthinking the whole matter, mate.

I am a bit high-strung. Nothing positive ever comes of fretting.

I just want to do things correctly.

What's the saying; anything worth learning is worth learning right

 

[Beezy]

Right, and we're telling you that this really isn't worth learning.

 

[Chris2412]

I appreciate all the input in this thread.

Perhaps when I am more educated I can ask the "hard questions", instead of running around like a rookie designer with his head chopped off.