Bots are giving inaccurate thread view numbers

Discussion about this site, including these forums (eg, suggestions, comments, queries). Topics may be manually deleted occasionally (eg, after suggestions dealt with, or changes bedded in).

Bots are giving inaccurate thread view numbers

Postby commando » Thu 28 Jan, 2021 10:32 pm

Bots seem to be the only ones scouring the forums and adding unrealistic thread count numbers, thousands and thousands more.
EG. Bing [Bot], Google [Bot], J M, Majestic-12 [Bot]
Privacy among members is compromised
commando
Athrotaxis cupressoides
Athrotaxis cupressoides
 
Posts: 413
Joined: Tue 14 Jul, 2020 10:32 pm
Region: Other Country

Re: Bots are giving inaccurate thread view numbers

Postby CasualNerd » Fri 29 Jan, 2021 8:29 am

If the forum software is picking up the bots as it should, I don't see why it would be adding those numbers to thread counts. The thread counts all look pretty reasonable too.

What's the privacy concern ? It's a publicly available website.
User avatar
CasualNerd
Athrotaxis cupressoides
Athrotaxis cupressoides
 
Posts: 481
Joined: Wed 03 Aug, 2011 3:33 pm
Region: Tasmania
Gender: Male

Re: Bots are giving inaccurate thread view numbers

Postby FatCanyoner » Fri 29 Jan, 2021 10:15 am

As someone who runs a different, smaller forum, I can tell you that all those bots are search engine crawlers. You don't see them do their work on most websites, but in forum software they are generally given specific privileges. That allows the admins to control what parts of the forum are added to search engines, and what areas are not captured. It also means their work is transparent. Search engine bots don't impact thread counts. They are important though, because without them people wouldn't stumble on this forum based on web searches. There is absolutely no privacy issue, because all they trawl is public posts.

There are many nasty bots out there, but they won't show up because the nefarious ones don't get access to those special privileges. They generally just try to infiltrate the site as standard members then post spam. I guarantee the admins are kept busy keeping them out. They could potentially skew the thread counts, but they mostly just try to post to forums.
User avatar
FatCanyoner
Athrotaxis selaginoides
Athrotaxis selaginoides
 
Posts: 1021
Joined: Fri 12 Aug, 2011 7:45 pm
Location: Blue Mountains
ASSOCIATED ORGANISATIONS: www.FatCanyoners.org www.CanyonGear.com.au
Region: New South Wales
Gender: Male

Re: Bots are giving inaccurate thread view numbers

Postby wildwalks » Tue 09 Feb, 2021 8:35 pm

Thanks guys.
Yes I think I agree with FatCayoner and CasualNerd.

This is a public website, so people are aware (and it is a good thing) for our conversation to be found and shared.

Search engine Bots don't keep smashing a page, and we do detect common bots and ignore them in counts.

There are occasional bots that are just roaming the web looking to cause trouble that looks more like regular users -- but if this was the case then the count would be more evenly spread across forums.
Instead, I think what we see are actual numbers -- the do correlate fairly well with Google Analytics (which is very good at filtering out non-human users).
In the last 12 months, we have served about 1.5m pages. Most of our users are stalkers, not members :)

With really good and a very long history of information (and with many forums and FB groups now hidden away from search engines) the information in this forum actually rates very well. I see spikes in usage correlating with something in the media or sometimes changes in search algorithms meaning pages get promoted.

So I recon we are doing something right. We are a helpful community making sure that we responsibly share our ideas and experiences broadly.

Thanks
Matt :)
wildwalks
Magnus administratio
Magnus administratio
 
Posts: 897
Joined: Mon 22 Nov, 2010 4:35 pm
ASSOCIATED ORGANISATIONS: Wildwalks, Bushwalk.com & NPA NSW
Region: New South Wales
Gender: Male


Return to Forum & Site

Who is online

Users browsing this forum: No registered users and 7 guests