Admins: Instnace randomly running extremely slowly? Check for this

submitted by
edited
35
130

Log in to comment

Hot Top New Old
edited

I found the same IPs doing the same thing for my server, but one thing I noticed in the access log was that nginx was returning a 499 status code. That code means that the client closed the connection before the server answered the request. So this seems to be a deliberate attack instead of the rash of bots many have been dealing with recently. They just firehose out requests to DoS the server since pagination on services with dynamic data is expensive.

I ended up creating a fail2ban rule to add any IP to my firewall blocklist that makes a bunch of 499 entries.

Edit: I also set a rate limit in nginx for any url that has a “page” query included

 reply
22
OP

Good idea with the f2b integration.

I thought about that before just blocking unscoped requests to that endpoint in Nginx.

 reply
7
OP edited

Can’t edit the post (Thanks Cloudflare! /s) but additional info:

  • I truncated the log excerpts in the post. The user agent string in these requests isn’t shown here, but it is blank in the actual logs.
  • This is for Lemmy admins only. It might apply to others in some form, but this seems to be specifically exploiting a Lemmy API endpoint
  • My Nginx solution may have room for improvement; I was just trying to block that behavior without breaking comments in posts and move on with my day. Suggestions for improvement are welcome.
 reply
21

I am gonna try to make it for caddy too

 reply
5
edited

Get a blocklist and set it up.

Literally all of the IPs are known bots for up to 3 years:
* https://www.abuseipdb.com/check/134.19.178.167
* https://www.abuseipdb.com/check/213.152.162.5
* https://www.abuseipdb.com/check/134.19.179.211

Oh and maybe also a rate-limiter…

 reply
15

That “jackass” sounds like an AI training set scraper. They’re known for being incredibly brutal to the sites they scrape, ignoring robots.txt and other honor-based systems for preventing the site from getting overloaded.

 reply
29
edited
 reply
9

I found that the Caddy handler above blocked many third party clients and even Tesseract.

So instead I’m using this CEL expression to return a 444 error on match of the unscoped old-sorted 50 per-page comments past page 99:

@block_comment_spam expression <<CEL
    {http.request.uri.path} == "/api/v3/comment/list" &&
    {http.request.uri.query.limit} == "50" &&
    {http.request.uri.query.sort} == "Old" &&
    int({http.request.uri.query.page}) > 99 &&
    {http.request.uri.query.post_id} == ""
CEL

handle @block_comment_spam {
    respond 444
}
 reply
6
edited

FYI these are all on ASN 49453

The other (lazier) option is to block/challenge the ASN

 reply
9
OP

That’s my normal go-to, but more than once I’ve accidentally blocked locations that Let’s Encrypt uses for secondary validation, so I’ve had to be more precise with my firewall blocks

 reply
6

Good, I am challenging all ASN 49453

 reply
1

Sounds like such unscoped requests should not be allowed in the first place? Maybe worth reporting in a Lemmy issue?

 reply
9
OP

That was my thought, but also wasn’t sure since there might be a use-case I’m unfamiliar with. I vaguely recall seeing a feature request for Photon a while back to be able to just browse comments, so I assume that would be how it worked.

But yeah as it is now, it can be abused.

 reply
7
edited

It’s only useful with the ModeratorView type. I haven’t heard more than just a few using it for anything other than for moderation purposes.
It is useful for some type of bots, for example.
But I think they should opt in for a solution with the upcoming plugin system (for example a webhook). Polling this endpoint is not very efficient and it is very possible to even miss some comments.

So I think this endpoint should be just for the modview type and authorization should therefore be required.

Or the rate limit should be more fine tunable. There are like 4 configurable rate limits that encompass all endpoints.

 reply
5

This is for Lemmy I presume (or also for Piefed or Mbin)? You’ve modified yours heavily though, I thought, which could complicate matters. I wonder if you are having those bot scraping issues that semi-recently (a month or so ago?) started increasing in frequency. So many instances now have a human detector before letting you in whereas before it was not necessary.

 reply
7

PieFed has a similar API endpoint. It used to be scoped, but was changed at the request of app developers. It’s how people browse sites by ’New Comments’, and - for a GET request - it’s not really possible to document and validate that an endpoint needs to have at least one of something (i.e. that none of ‘post_id’ or ‘user_id’ or ‘community_id’ or ‘user_id’ are individually required, but there needs to be one of them).

It’s unlikely that these crawlers will discover PieFed’s API, but I guess it’s no surprise that they’ve moved on from basic HTML crawling to probing APIs. In the meantime, I’ve added some basic protection to the back-end for anonymous, unscoped requests to PieFed’s endpoint.

 reply
4

Could you elaborate on this:

it’s not really possible to document and validate that an endpoint needs to have at least one of something

In what sense it is not possible, as I can easily see it done in the code?

 reply
1

It’s straight-forward enough to do in back-end code, to just reject a query if parameters are missing, but I don’t think there’s a way to define a schema that then gets used to auto-generate the documentation and validate the requests. If the request isn’t validated, then the back-end never sees it.

For something like https://freamon.github.io/piefed-api/#/Misc/get_api_alpha_search, the docs show that ‘q’ and ‘type_’ are required, and everything else is optional. The schema definition looks like:

/api/alpha/search:
    get:
      parameters:
        - in: query
          name: q
          schema:
            type: string
          required: true
        - in: query
          name: type_
          schema:
            type: string
            enum:
              - Communities
              - Posts
              - Users
              - Url
          required: true
        - in: query
          name: limit
          schema:
            type: integer
          required: false

required is a simple boolean for each individual field - you can say every field is required, or no fields are required, but I haven’t come across a way to say that at least one field is required.

 reply
1

Ah, I see, so you are talking about this.

Of course it is nice if things get auto-generated, but doing it yourself, both in code and documentation should never be excluded as an option.

 reply
2

Exactly that, yeah. Thank you for the link.

 reply
1
OP edited

Lemmy. I added a comment above since LW wouldn’t let me edit the post.

Mine’s only extended with some WAF rules and I’ve got a massive laundry list of bot user agents that it blocks, but otherwise it’s pretty bog standard.

If instances have Anubis setup correctly (i.e. not in front of /api/...) then that might not help them since this is calling the API endpoint.

 reply
4

All of a sudden your edits went through - perhaps a delay caused by this same issue?

Also some related posts:
* another one reporting similar attack-like activities https://lemmy.world/post/36413045
* a month ago similarly https://lemmy.world/post/34310429

 reply
3

Maybe we should introduce a gated API and charge $12 for 50k requests…

 reply
6

We had this issue on and off for a few weeks at least, causing massive postgres CPU spikes. I ended up blocking large page params with an nginx regex.

 reply
6

Isn’t the same endpoint used to list all of a user’s comments in their profile?

 reply
5
OP

No, that’s just /api/v3/user which returns both posts and comments.

 reply
5

Things have been slow for me off and on in recent weeks. And today it’s quite slow.

 reply
3
OP

Unfortunately, there’s many many reasons that could be the case. I’m just putting this out there since it’s easy to check for and mitigate against.

 reply
3

Not sure if there’s a legit use for just fetching only comments outside of a post

The ability to see all comments is right there at the Lemmy UI.

 reply
3

To everyone in this thread, if you notice a problem in Lemmy please open an issue. We are only two developers and dont have time to browse the Fediverse all day to come across such things. Only if we know about a problem can we actually fix it and make a new release.

For reference here are the issue and proposed fix:
- https://github.com/LemmyNet/lemmy/issues/6016
- https://github.com/LemmyNet/lemmy/pull/6017

 reply
2

I think this could end up blocking https://lemmy-meter.info/
They make requests without post id

 reply
1
Insert image