Super slow and not loading

[caddisguy]

Super slow and not loading or server down off line notifications

What was the last time you encountered this? Should be fixed as of about 10PM. Been clicking around like mad and no longer able to replicate the issue since I blocked the bot net.

[Husky7mm]

What was the last time you encountered this? Should be fixed as of about 10PM. Been clicking around like mad and no longer able to replicate the issue since I blocked the bot net.

That was a few hours ago. It seems good now, you the man! It started yesterday mid day. I work in the bush so for a bit I thought it was just the service.

[caddisguy]

That was a few hours ago. It seems good now, you the man! It started yesterday mid day. I work in the bush so for a bit I thought it was just the service.

Awesome news there! Thanks. Looking at the logs it looks like it's been off and on a bit since the 13th. Hopefully smooth sailing now but please let me know if you run into it again.

[Spy]

Good stuff seems to be back to normal. Thanks Cadis

[caddisguy]

Good stuff seems to be back to normal. Thanks Cadis

You're welcome. Thanks for creating the thread!

If you or anyone is curious, the bot net that was hitting us is ByteSpider / ByteDance which are run by the same Chinese company that made TikTok. It is very suspicious as to the reason they are scraping data from from all over the Internet. It's doubtful they are setting up a search engine. Pretty sure this website would not be allowed in China anyway? Some speculate it is for AI training but for all anyone knows, it is possible that it could be for looking for security vulnerabilities on a wide scale that could be exploited at a later time.

Their robot crawlers are particularly aggressive. It seems continuous and not throttled in any way. They also do not obey the traditional robots.txt file (which is a standard for instructing crawlers which ones are allowed/disallowed and what content is permitted to be crawled) but since they all identify with the same UserAgent string I created an .htaccess mod_rewrite rule to block them by their agent ID.

RewriteCond %{HTTP_USER_AGENT} ^.*(Bytespider|spider-feedback|bytedance).*$ [NC]
RewriteRule .* - [F,L]

If they change their UserAgent string, I'll do something more fancy

Here's a YouTube video someone made about these ByteSpider / ByteDance robots.


https://www.youtube.com/watch?v=Hi5sd3WEh0c

[Spy]

Should we be worried in any way? Is it collecting info on members? Just asking if we need to be changing passwords ect.

[caddisguy]

Should we be worried in any way? Is it collecting info on members? Just asking if we need to be changing passwords ect.

No we're all good there. They were not making any POST requests to try to exploit or break into anything at this time, so they can only see what the rest of the public can see on this website.

The only difference between a public visitor browsing around on the site and this bot net, is that they were able to read and most likely copy (because that's the whole point) a good chunk (maybe most) of the publicly viewable posts on this website in a matter of a couple days. I know from 4AM to 11PM yesterday, they read (and copied of course) over 500000 pages. Later on I'll go back to the 13th (looks to be when it started) to see what else. If I recall, the site has around 2000000 posts and there are more than 4 posts per page, but of course there are other things we can click on besides posts, such as users profiles and such.

They won't have anyone's passwords and can't read PM's and such, but if your profile picture shows up on a Chinese cereal box or in an AI video, we know who did it.

This all isn't anything particularly new. As we know, the public internet and social media is just that. Anything we post publicly can be read or recorded and used for beneficial things (like legit search engines) or others who benefit from it in self serving ways that are not helpful to us.

I should also add these ByteSpider / ByteDance bots are crawling virtually the entire public Internet. HBC was not specifically targeted. It just happens to be part of the public internet (at least the parts of it that are publicly visible)