• hperrin@lemmy.ca
    link
    fedilink
    English
    arrow-up
    3
    ·
    11 days ago

    I host tons of open source projects on https://forge.sciactive.com/

    Recently, it was getting absolutely hounded with tons of meaningless requests. I’m assuming they are just results of AI.

    So, I put Anubis in front of it. Now images hosted there won’t load on other sites, people have to wait forever to see the code, and the experience on phones is even worse. Just because AI doesn’t respect anything, and the people who run it are too stupid to realize that.

    This is why we can’t have nice things.

    • MonkderVierte@lemmy.zip
      link
      fedilink
      English
      arrow-up
      2
      ·
      edit-2
      11 days ago

      You can set a trap that isn’t in robots.txt or is forbidden there. Anyone who ignores your robots.txt is not friendly.

    • AHemlocksLie@lemmy.zip
      link
      fedilink
      English
      arrow-up
      2
      ·
      11 days ago

      Just because AI doesn’t respect anything, and the people who run it are too stupid to realize that.

      They’re perfectly capable of realizing it. How do you think it got built that way? They don’t respect anything, either.

    • Moop@lemmus.org
      link
      fedilink
      English
      arrow-up
      1
      ·
      10 days ago

      It’s odd, I see so many people complain about Anubis taking a significant amount of time, but I’ve only ever had it 1-2 seconds on my devices

  • papercrane@lemmy.zip
    link
    fedilink
    English
    arrow-up
    1
    ·
    11 days ago

    It’s hard to post original work to share but also know it can just get scraped but I can’t give up personal websites. Social media looks so bland and seeing completely personalized blogs are the most enjoyable part to the internet now. Spending more time in niche spaces on the internet more than ever.

  • dieTasse@feddit.org
    link
    fedilink
    English
    arrow-up
    1
    ·
    12 days ago

    Yeah, internet is no longer the interesting place it used to be. Interesting, intriguing, exciting. Now it’s worse than going to a tax office… the day before taxes… On Christmas… 😂

    • kalpol@lemmy.ca
      link
      fedilink
      English
      arrow-up
      1
      ·
      11 days ago

      We should all go back to HTML and web rings

      And Usenet but with spam filters somehow

      • dieTasse@feddit.org
        link
        fedilink
        English
        arrow-up
        1
        ·
        11 days ago

        I am actually trying geminispace. It has nothing to do with google btw, but it’s quirky, it’s weird and I am actually getting some old internet vibes

        • andioop@piefed.social
          link
          fedilink
          English
          arrow-up
          2
          ·
          2 days ago

          For onlookers:

          https://geminiprotocol.net/

          Gemini is a new internet technology supporting an electronic library of interconnected text documents. That’s not a new idea, but it’s not old fashioned either. It’s timeless, and deserves tools which treat it as a first class concept, not a vestigial corner case. Gemini isn’t about innovation or disruption, it’s about providing some respite for those who feel the internet has been disrupted enough already. We’re not out to change the world or destroy other technologies. We are out to build a lightweight online space where documents are just documents, in the interests of every reader’s privacy, attention and bandwidth.

  • Schwim Dandy@piefed.zip
    link
    fedilink
    English
    arrow-up
    1
    ·
    12 days ago

    If I didn’t enjoy the cat & mouse of it, I’d definitely walk away from running a site. Just out of curiosity, I created an hourly graph today showing malicious bot, legitimate bot and human ratio graphs for one of my sites. The malicious hits make up around 80% hourly and human hits are usually well under 10%.

    image

    • bruh@thelemmy.club
      link
      fedilink
      English
      arrow-up
      0
      ·
      10 days ago

      What sorts of measure you have taken to prevent bots from exhausting your compute and bandwidth

      • Schwim Dandy@piefed.zip
        link
        fedilink
        English
        arrow-up
        2
        ·
        edit-2
        10 days ago

        As soon as an IP is determined to be malicious, subsequent pageloads result in a single query to match the IP and they get dumped to a 403 static error.

        With the bots that hammer a site looking for the top 1,000 or so exploitable URLs or partake in speed scraping, this has worked wonders for my server load. I catch them in their 1st-5th page load and the rest just result in basically no effort on the part of the script.

        image

  • Zak@lemmy.world
    link
    fedilink
    English
    arrow-up
    0
    ·
    11 days ago

    I host a few sites and I haven’t had a problem yet. Sure, logs suggest a majority of traffic is bots, but the cheap VPS I’m hosting on hasn’t melted so I haven’t been motivated to do anything about it.

    • Dave@piefed.nz
      link
      fedilink
      English
      arrow-up
      2
      ·
      11 days ago

      Lemmy is particularly susceptible. Hosting a static site with a limited number of pages is fine. But Lemmy needs to check all posts, find the votes, sort them, and compile the page. It does that for practically every page load, and so loading a page once is quite “expensive”.

      Then add on to that the almost unlimited pages of content for bots to crawl through, bots just get caught endlessly scraping pages. Even powerful servers struggle, which is why most Lemmy instances have implemented some sort of protection against bots.

      • frongt@lemmy.zip
        link
        fedilink
        English
        arrow-up
        0
        arrow-down
        1
        ·
        11 days ago

        It doesn’t need to. Like any similar site, it can cache a lot of that in databases like redis, or cache the rendered page in part or in whole.