Wholeheartedly agree. It's sad to slowly see the bitrot in my own bookmarks, trying to pull up important resources or media from only a few years ago only to hit a 404 or domain placeholder. And many times there is nothing on Archive.org or similar.
That ArchiveWeb extension looks excellent for custom captures. Picking the pages and elements you want during capture is a great feature.
For quicker one-offs, SingleFile gives you a single HTML with all the media of a page: https://www.getsinglefile.com
If you're making archiving a more involved part of your workflow, I highly recommend ArchiveBox. It can save pretty much every type of file from everywhere, in almost every format you can imagine, while building a searchable index of the archives. Been using it successfully for many years: https://archivebox.io
I just started using it, and to my surprise it only saves the sanitized html content, no scripts no css. I'm thinking of pairing this with archivebox. But hister is amazing. I love it. It's from the creator of searxng if I'm not mistaken which is another piece of great software.
Can vouch for Hister! In the fairly short time I've been using it, I already had a few sites disappearing or changing content and hister kept the juicy information accessible.
It was painless to set up and quick to use when needed, too.
I also use Singlefile and Obsidian Web Clipper a lot, but Hister has been great and is now a first-class citizen in my system. It helps it works passively for those things that don't look important when you read them, but become relevant later.
I am using both Linkding and Linkwarden self-hosted and archiving my bookmarks as HTML, PDF, and screenshot (I might be overdoing it) without the need for any additional dependency like client software or extension. Linkwarden works better at archiving things like Reddit, but now with the anti-scraping mechanisms in place it might het harder to do anyway.
I realized I really need this when one website which was a treasure trove for PowerMac technical information and assembly guides just went dark. Some of that info I could never find elsewhere. Linkrot sucks if you're in the middle of an (admittedly drawn out) personal project and your source of information vanishes with no trace. Belt and suspenders from now on. Might give Hister a try.
I used ArchiveBox for years. It's really good. The self-hosting and maintenance gets annoying after a while though.
Link rot really bothers me and I want people to get into archiving web pages so that eventually there will be a decentralized mirror of the unslopped web sitting on people's hard-drives, ready to share whenever someone needs it. So, I made a worse version of ArchiveBox ( https://memoryhole.app ). My philosophy is that, if people see how it is easy to start archiving and how useful it is, then they'll keep it up with SingleFile, or ArchiveBox or anything else.
I've been using Archivebox for years also, but it recent years it started to have multiple issues. My queues are full of pages that failed to scan, and it's been several versions since I actually had anything new there.
I did small amount of troubleshooting a few times with no resolution, but now I'm thinking of switching my thought process. Maybe something like a "web clipper" is much better, as complex SPA pages often fail to archive well anyway. And I should focus on content itself, not the page. That SingleFile extension fits well here as well.
It's funny - I actually went through a similar transition and also realized it was a little too heavy-duty for my personal workflow.
At this point I'm using Linkding (self-hosted bookmarks, which has an archive function) paired with Singlefile (which integrates with Linkding) which locally captures and uploads. For larger, or multi-page, captures I still fire off an Archivebox command. I run into blocks occasionally, but it also just feels like the open web is becoming less and less open in general.
> At this point I'm using Linkding (self-hosted bookmarks, which has an archive function) paired with Singlefile (which integrates with Linkding) which locally captures and uploads.
I second this workflow, exactly what I switched to from Archivebox. Linkding has a built-in "create HTML snapshot" feature but the Singlefile browser extension works much better.
Have you tried ArchiveBox since v0.9 was released? v0.9 was basically a complete rewrite with a new archiving engine, and it ships with ArchiveWebPage built-in as one of its plugins now.
As of the latest v0.9.77 it's now tested to be able to handle 2 million snapshots / 2TB on a 4CPU server, archiving @ 1 URL/min consistently while serving all the web UIs @ ~100ms. Also has automatic importing from social media, google docs, notion, canva, figma, and a growing list of sites with self-healing AI scripts & cookie sync from your browser so it doesn't need as much babysitting.
Nowadays it's Obsidian and Obsidian Web Clipper browser extension. The only caveat is what doesn't save the images locally by default and needs some manual action: https://obsidian.md/help/web-clipper/capture
SingleFile is great for single pages. All the tweaking I do is enable or disable saving scripts, and it works for >90% of webpages. Output being a single file (heh) simplifies things enormously, there’s no hierarchy or folders to deal with - hard to imagine your saved files not loading or breaking in some way as long as the original capture works.
I haven't tried archivebox, but self-hosted https://karakeep.com/ has been working well for me. When the server-side parsing fails I'll fallback to using SingleFile in my browser which then forwards it to my Karakeep instance.
I use AnyBox on macOS to do this for bookmarks. It saves a WebArchive copy and a stripped 'reader'-esque version. SingleFile can also pipe into AnyBox, if you want to use that as the archive.
Archivebox annoys me because (as far as I remember) it doesn't have any options for capturing a webpage on your end and uploading it to your self-hosted Archivebox instance. With numerous sites moving to Cloudflare to mitigate AI agent crawling, it makes it really hard (or outright impossible) to capture a page because it wants to do it server side.
The ArchiveBox Browser Extension https://extension.archivebox.io/ has this built in. It captures both full MHTML and screenshots and uploads it to your server automatically, it was added a few months ago.
Also works from iOS and Android so you can save logged-in content directly without having to set up cookie sync (which it also supports now too directly from the extension).
This is basically the main reason that I started building https://github.com/mfinelli/recueil. It _only_ works in your browser so you can bypass captchas or paywalls or whatever and then it uses singlefile (https://www.getsinglefile.com) to upload the result to your self-hosted archive.
I’ve been going over the art that I’ve saved in the past 15+ years and it’s quite sad when I go look for the original source of an image and find out that it’s completely gone from the original website. No indication that they’re on some other website, at least by reverse image searching it. Definitely has made me glad that I’ve been hoarding these images on my hard drives, with the total count being somewhere over 300k at the moment. I think the oldest ones have now been migrated across four different computers. Kinda sad for everyone else since they can’t see them anymore though.
Haven’t started saving web pages yet, but I definitely should though. My home server has some tens of terabytes to spare, so that’s at least not an issue. Been eyeing these comments to find a good self-hosted solution.
On MacOS, I use my own built app called HEAP which lets me archive in various formats locally (full page image, pdf, markdown, html, and webarchive) via a single right click of the menu bar button:
In 5 years, cloud LLMs may have largely replaced the open Internet as a source of information, with centralized control and ability to rewrite the past.
Please, please buy books and keep them in good condition, and please archive websites with important information (assuming it's legal in your jurisdiction). There are still plenty of excellent sites and blogs around from the 2000s and even earlier, but they won't be there forever. Ironically, coding agents have made archival much easier -- I've always found problems with using pre-written tools, but Claude or Qwen make it trivial to download, test, and archive sites on your local.
I don't think that mainstream models will try to rewrite the past, but I wouldn't trust models to helpfully engage with your intent. As search engines encode better understanding of queries, model-like recalcitrance for "naughty" queries becomes easier to implement. Just compare something like
Disclosure Day free stream
across Yandex and Google. I'm sure that Google has indexed the same web sites, but Yandex has a free stream as the #1 result and Google has either deindexed it completely or pushed it below dozens of less-relevant, more-legal sites. (According to Google, the site Yandex showed me wasn't even DMCA'd out of the results.)
This isn't just relevant for piracy. Paid subscriptions to frontier models will refuse or downgrade answers to questions about the natural sciences and engineering. Unlike cybersecurity, where you can get specially approved to use models with fewer refusals, I don't think there is any program you can join where Anthropic will treat you like a grownup who has already read Carleone's Tactical Missile Warheads book. Opus just won't discuss details of high explosive engineering at all. It's tolerable as long as search engines still surface relevant documents that you can read with your own human eyeballs. If LLMs devour the market for search, it's going to become alarming.
Not the provider per se, but there have been several documented attempts to rewrite Wikipedia around historical and geopolitical topics. One of the reasons for this is Wikipedia feeds the chatbots.
There are already many people who are trying to rewrite the past. Holocaust deniers, Moon Landing deniers, people who deny the US Civil War was about slavery, and so on, and that’s just one country. Give one of them the ability to put their thumb on the scale when training an LLM and they absolutely will.
They been doing this from the start. Remember when Google's new image generator model, when asked to generate pictures of historical white people, would render them as black without fail? We only know of that example because of how incredibly ham-fisted and incompetent it was. The future of AIs is likely for them to only become more contaminated with ideological or monetarily incentivised biases, and we'll be less and less likely to be able to spot them.
It depends on what you mean by 'an LLM will try to rewrite the past'. But in either case I don't believe that stating that an LLM will try to rewrite the past is not an obvious, deliberate exaggeration (ie, hyperbole).
If you mean 'an LLM' as in the owner/maintainer of the LLM/weights, then examples of this already exist. Whether it's DeepSeek/Gwen downplaying Uyghur forced labor camps [0] or Grok hallucinating a 'white genocide' in South Africa [1], it is clear that the owners of LLMs have political agendas that are expressed in the weights of popular LLMs.
On the other hand, if you mean 'an LLM' as in an autonomous agent that could decide to rewrite history, that also seems very possible. Either by using existing techniques like, "poisoning ChatGPT, Gemini, and Google AI Overview answers by seeding the Web with malicious links and data and then optimizing the content." [2] Or by breaking out of its sandbox and updating its own internal weights that allow for an outcome that distorts historical facts. "In July 2026... OpenAI models circumvented controls designed to isolate them from the internet and compromised parts of OpenAI’s internal research infrastructure and Hugging Face’s systems ." [3]
I learned this the hard way by getting too casual with Angelfire keeping an old website running for 20 years after I stopped updating it.
It had lots of photos that I hand-scanned from printed film, back in the day when it took a good bit of time to scan photos manually.
The site was a running joke, where I would open it to show people that it still existed... until it didn't, and I couldn't find my backup of all of the photos. The archive sites have the 'bones' of the website, but not the photos. It was easily avoidable, which is why it is painful.
Ooph. This is what I’m using netlify for. I clones all my old php sites into static copied and moved them to Netlify. Hopefully I’ll get a heads up before they close.
This made me realize that I don't actually love any website like that. Wikipedia comes the closest, but even if that would disappear, surely someone would have a dump, which I'd grab at that point. Until then, I won't do this.
The two things I go back to are music, and books. Even books not so much, but songs tickle a very specific part of my brain, that I want to reproduce over and over again, and then not for years, and then again. And when I want to go back and I can't, and I know I could, that really hurts, and I feel betrayed. So I do actually archive the music, even from streaming sites.
I have entire albums in my old iTunes library that you literally cannot find online, anywhere. Hip hop beat tapes that were uploaded on a music blog in '07, and the blog was behind a Blogspot adult notice and so it never got archived. Or old PHP-based music forums that did URLs via query param, which only ever got their homepage archived. Old James Blake demos from Dubstepforum whose Rapidshare links have been 404ing for 15 years.
The pace of bitrot is freaking crazy!
Even my library is not what it was, I had two drives fail on me the same day in college and I lost over 100GB of meticulously-tagged 320k and FLAC files.
Wow, that's impressive. Have you considered uploading them somewhere so that their access via the internet can be restored? I'm sure someone would be interested in what you have to share and they are a precarious "bus factor of 1" away from never getting the chance.
Yeah, it's crazy how music vanishes. Well anything really, from the internet, since keeping it up needs active work on someone's part.
I too have a library that's somewhat in disrepair. In my case, a corruption happened when I was hauling my hard drive around - and it didn't happen when I chucked the hdd in my bag, it happened when I have set up the wrong sector and header information in my buddy's bios! Suddenly the chemical brothers songs had some black metal in them.
This is timely for me, since a web forum I care about went offline, and I simultaneously discovered the Wayback Machine for whatever reason never crawled most of it. I asked Claude to try to put together a static site of what could be salvaged from various archives and other sources, but as miraculous as what AI can do these days, it can't restore data that is truly gone - so what it was able to salvage is better than nothing but still missing a lot.
I know that feel, bro. Wayback Machine is very spotty, unfortunately. websites that existed for years often have only a few pages archived.
the worst thing I saw was a website that was dead for over a decade but fully archived get hit with "Sorry. This URL has been excluded from the Wayback Machine" because of robots.txt set by whoever is squatting the domain now. I've managed, by some miracle, find an archive of its content on some random forum. bookmarked it. by the time I'd remembered about the bookmark, that random forum was dead and the link was not archived :(
After losing the Japanese indie web of the 00s/10s several times over, I wish it was possible to save not just the websites you love, but the websites you'd love if you actually knew about them. So often do I find out that what I was looking for did exist, but doesn't exist anymore. In particular when only the home page got archived, but not the rest of the website, nor the uploaded files.
I've backed up lots of CS courses, you know those old html websites written by the professors by hand. `wget -mkpnp` works very well for those. Even years after finishing uni, I've used those websites for studying for interviews.
Too bad most of them are now gone.
Bonus suggestion: get that internet archive extension for your browser that automatically sends every page you visit to the IA (if you're ok with the privacy side).
You don't have to worry about having your copy easily accessible, and as a bonus it's also available for everyone else in the world.
I recently spent a few days trying to figure out and discover my old websites and things from the early 2000s on the web archive. I hadn't realised my memory had gone quite so bad, but it was revitalising to find some of what I'd forgotten. So much of what I do remember is long dead and missing, sadly.
I thought it might be rose-tinted glasses, thinking that the earlier internet was better, but spending time retracing my teenage steps made me even more certain that it was indeed better. So many derpy little websites made by nerds, so many fun games and websites that weren't preying on vulnerable people, and everything spread out across many places - not centralised in the hands of large corporations. I've been using LLMs to have fun recreating concepts from that time, like my own forum software, as some sort of anti-progress. It's been fun.
I remember an old niche website I saw some 20 years ago. The whole content was dynamically loaded. If you disabled javascript, you were greeted with a static page that accused you of theft. Attempting to select and copy text or right click resulted in a popup window saying the same.
Nowadays you can just slap your site behind Cloudflare, I guess.
Nowadays you can open the dev tools before loading the site and then select or copy whatever you like in it.
But yeah some websites were really special. I remember one made by some Microsoft fan which was completely unusable outside Internet Explorer. It was like a static picture in any other browser, clicking links & buttons just didn't do anything.
Video backups are also important too as they can be easily censored when within a walled garden.
Interestingly, "there is no single direct 1:1 Linux clone of NirSoft’s VideoCacheView that automatically monitors browser cache folders via a graphical interface. Released 2007." https://videocacheview.en.lo4d.com/windows (Gemini)
It's not really clear to me why people are proposing so many complex approaches or third-party solutions here. What exactly am I missing, what's insufficient about my browser's "Save page As... -> Web Page, complete" functionality?
I've been using self-hosted Linkding for about a year now to save bookmarks. It will also use a Chromium instance to scrape and archive the content/page in an HTML format.
I've not had to use that yet, but it's also nice to save terms and conditions and the like when you agree to them so there's no sneaky edit.
Agree with this. I started down learning more about archival while I was adding it to my own thing [1]. I wish there was more of an existing toolset for diffing archivals as well - what changed is just as interesting as what is there. Equally, something I kind of think is lacking in WARC/WACZ is exploration of the archives - maybe forensics is too strong a word but it would be nice to have an easier way to explore what you’ve captured and understand the data in different dimensions.
I’ve supported some of this in my own tool, ZShot - capture in sqlite, Quicklook extensions for WARC and WACZ. Just trying to make archiving easy and interpretable.
It’s also been interesting to see people use archival but for different uses: Kiwix [2] showed up in my reading recently, “archival but for offline internet”, which I’m kind of fascinated by.
I have been using HTTrack is cases where I want a browsable local version of any site (not just a page)! Most of the applications mentioned here as comments only allow for saving a single page. But afaik, only HTTrack allowed me to save a local version of the website and not just a webpage. The “depth” argument allows me to do that when downloading a site through CLI
At the moment I use Obsidian and some CLI magic to extract my discoveries out of the Web into private notes. I call that My Own Private Wikipedia. Sounds like a good practice as things are organize the way I want, and I can rearrange in my own way and things are backuped as I wish.
the last source of truth are forums.. but not reddit... if it has a CEO it will rot and enshitify.
I'm talking about forums... the bmw forum for your old car... the motorcycle forum for the enthusiast of the SV650... those places are full of anecdotes, people talking, lending a hand to another person... being humans and care...
Which is what social media is/should/was about, is what internet is about before the enshitification happened... I would vote to have this ID associated to you via check of your local goverment issued id and ONLY UPPON VERIFICATION of you being using that you can log and read and use those websites without tracking you (Yes 10,000 ifs where this is secure, handled in a confidential matter, the whole tech is not to track that you are doing something... but that a human is behind that, no api, no robot, no adds, no nothing... just the old school forum where you can go, talk, submit a photo... where someone who wanted to spend time to write something did it, SO WE CAN ENSURE TWO THINGS, A HUMAN IS THERE, AND NO FVK!NG CEO WILL ENSHITIFY IT, no to track people, but to know that a person is interacting there)
But because "empathy is the weakness" and shareholder value...
> the last source of truth are forums.. but not reddit... if it has a CEO it will rot and enshitify.
Quick gripe-anecdote that occurred (again) just now: I have a technical question, the search-engine says it was discussed in a Reddit thread, so I go to Reddit, giving them my traffic and eyeballs just like they want... but the information is impossible to find because "new" Reddit hides/collapses 90% of the discussion by default and there's no way to tell which branch has what I need without manually clicking them all. (And them some of them will reload the entire page, erasing progress.)
So I have no sympathy for their whining about scraping when they spent already years making it painful for actual live organic humans to find anything.
> Which is what social media is/should/was about
I think our two imperfect approaches here are:
1. Webs of trust, in some way that resists crowds of sockpuppets.
2. Some kind or proof of stake. Proof of anonymous-humanity is hard, proof of unique-humanity is creepy spy stuff, but "proof if you ban me it'll at least hurt a little" might be easier to achieve.
"1. Webs of trust, in some way that resists crowds of sockpuppets." we are already there... is just that the frog soup had a very steady hand at rising the temperature.
The other option is to use the X402 protocol to pay, every time you access a page you pay in cpu to calculate some shitcoin.
For a person, is just trading the time spent loading and displaying ads for paying the creator for their work.
But if you are scrapping you are now paying with cpu and electricity to see things and content creators are being paid.
This is why I started using Obsidian Web Clipper. There is SOME work to learn it and set it up, but the small amount of upfront work will pay for years to come. And the folder itself is backed by Dropbox, so I don't have to worry about losing it all forever.
I also regularly use the web clipper specifically for this reason - most of my conversations can be around articles I read online, and instead of googling for the article I have it referenced right in my journal. I find myself regularly upset that I didn't save old articles from years ago because I didn't have this system
Or don’t. Lots of things I loved in the past and spent time backing up and backing up the backups that after 15-25 years I realized was a waste of time. Thought I had plans for them but never looked at them. Never referred to them. Finially realized I never was and got rid of the final backups
Yup! especially as we are using various ai platforms and ai agents.
I created my own artifact aggregator service which hosts the artifacts. I run it on oracle cloud for free and back them up on my gmail drive. https://simple-host.com, and you can back
I've tried almost all the popular and many not-popular options to no avail. Does anybody know of an archiver of any variety (Firefox extension, or CLI give-it-a-url, or anything else) that can archive a modern Reddit page with the voting (e.g. I can later see if I up- or down-voted a post)?
Seems like an extension an LLM should easily be able to make in a few minutes. Start with asking an LLM to use a Chrome MCP to see what it can do and if its feasible.
SingleFile should handle that without any issue. Or use client scraping option in Karakeep which integrates SingleFile into a self hosted Pocket replacement.
I've started to use a custom bookmarklet for this. If I like something I read or watched, I 'pick' it with the bookmarklet, add my thoughts, and categorise it. The script then checks if it's on the Internet Archive, if not it tries to archive the site, and then save it to my public repo, which then my personal website shows under /library.
Client scraping with SingleFile extension or Karakeep works perfectly on anything visible in your browser window from my experience. Including SPAs/JS rendering/aggressive CAPTCHAs/etc.
For CLI automation could probably one of various fingerprinting resistant Chromium instances with the extension setup in a Docker container or similar. Run from your home Internet for the residential IP.
Something I have been doing is saving PDFs of interesting articles. I actually go back for some of them. Organization is always an issue but mostly desktop search finds it.
This. Aggressive AI crawlers have ruined many parts of the web. I guess you have to ask the owner of the site nicely, if they would provide a complete copy of the site for your personal archive.
I did actually. I took a whole day to go to each of my individual saved bookmarks and save the articles as pdf using edge reader mode, sometimes hunting down articles on the way back machine
This has come up for me very recently with the multi-week AO3 outage. I'm not a huge user of that site, but it specifically existed as a comprehensive archive of fanworks and of many prior fanfiction sites that have gone offline. I really respected the Organization for Transformative Works as a non-profit with a very targeted mission of archiving these sorts of works and keeping them safe from linkrot and the algorithmic web. Now, after a security incident (which I am assuming is related to the recent LLM-driven discoveries of unpatched vulns in open source hosting software), the whole archive is offline for the next few weeks.
I'd always assumed that OTW would keep AO3 online until the end of time. It, somewhat like this site, was one remaining gasp of Web 2.0 before the algorithmic feeds took over.
I did this for a while. I went so far as to built a highly advanced web crawler webapp that was better than the Internet Archive in a lot of ways.
But the modern web has made this unreasonable. Every webpage is 99% junk (CSS, JS, HTML structure, tracking pixels, UI images, navigation links, cryptominers (I joke)) and 1% content. You need a smart AI system to pick out the good parts, otherwise you'll quickly find yourself schlepping hundreds of gigs of junk to just preserve your favorite websites.
The better idea is: if you love information, back it up. Collect the knowledge from the website, not the website. And then preserve it. Create a free ebook. Or sell it on Amazon kindle. That'll last a lot longer than the website will, and any HTML backup of the site you could possibly make.
I'd suppose most of those junk websites have no value in them, really. The most valuable information in the web is in good ol' (x)html, maybe some css. But something like readability.js [1] can help in filtering the junk around the main info.
Have a look at Hister [1], a 'personal web history search engine' which by way of extensions for browsers captures your web history and stores actual page content on your own instance where it can be browsed and searched. It does not store images so if that's what you're mostly interested in it won't work for you. For text is works quite well, maybe sometimes too well since it also stores ephemeral things like HN comment pages unless you tell it not to.
Plain text is not robust at all. Not now, and not in the futue. Its actually useless.
How do you save images in plain text ?
Lot of usefull information is stored in pictures.
ie: in technical articles about graphics, hardware.
Or some diagrams mentioned and refereced in the text.
Even medieval books have pictures, because some things need to be shown, and it is not enough to just write some text about it.
Anyway, youre probably just a very very old troll, typing this on a typewriter...
Obligatory plea: when scraping sites for archive purposes, take care to do so slowly so as to not impose an unfriendly amount of load on the target site. Otherwise you risk becoming part of the problem that is making some take down their resources ATM.
Yes, and with AI you can even back up the logic behind a website (the back-end), and even generate fake but plausible data to fill it if crawling is too much work.
It is kind of sad to see how many webpages vanished.
This includes old games. Many vanished. Some made a
cameo, coming back, but it is often no longer the
same. Case in point: The Reincarnation (TR), as a follow-up
to the old Mari Archmage (AM) game. It was great in the
late 1990s. Mari went bankrupt, TR came about and improved
on many things, but also made a few things worse. Player
population declined, then they had that genius "idea" to
add bots. That was when I retired. I can not want to be
bothered to "interact" with code.
Agreed. I won a major (at least within the college) music competition at a local college like two decades ago, and now all traces of it have vanished. If I had the URL I might be able to pull it up on the Internet Archive.
I had a graphing calculator website that had 250K monthly active users, from the logs it seemed mostly schools' math classes in Latin America. It was running on a single AWS EC2 Classic.
Ironically, when I started working for AWS, I always felt threatened with sticks and never had space to maintain my own projects. When they deprecated EC2 Classic, my site went down. I didn't get time or mental space to migrate it.
A website I miss was a bullshit corporate jargon generator from Scott Adams. There were 3 columns with corporate jargon. And every time you refreshed the page, a new 3-word phrase would be highlighted. They were the most ridiculous phrases. But one could imagine them being used.
There are far better tools that are not built by this person’s company. Dig around and you’ll find an incredible array. Not everything has to be a sales pitch.
Ages ago (like 2001 or so?) I started building something to automatically archive everything I browse. I used tcpflow (https://www.circlemud.org/jelson/software/tcpflow/). I've seen a fair bit of varions of this idea over the years
Then SSL/TLS started getting widespread and I lost focus. Nowadays, I guess you would build it as an extension in the browser.
i'm saddened that http://macsecret.com/ along with /files directory seems to have all be been lost. I has waiting on some storage for the NAS to archive it and it went away before I could back it up.
God forbid we have an AI-facilitated full Cyberpunk/.hack-style crash where everything gets wiped, the only things that will survive will be on paper. This is the recommendation I give my friends who are digital artists/writers; have a physical record of your efforts beyond what you want to sell.
This is on top of keeping your own digital archives. I'm still haunted by the deviantArt submission I didn't save. The creator took the piece down, then passed away a few years later. A Wayback Machine archive of his userpage with the piece as his featured upload still exists, but the image itself wasn't archived; it just shows a broken image.
Wholeheartedly agree. It's sad to slowly see the bitrot in my own bookmarks, trying to pull up important resources or media from only a few years ago only to hit a 404 or domain placeholder. And many times there is nothing on Archive.org or similar.
That ArchiveWeb extension looks excellent for custom captures. Picking the pages and elements you want during capture is a great feature.
For quicker one-offs, SingleFile gives you a single HTML with all the media of a page: https://www.getsinglefile.com
If you're making archiving a more involved part of your workflow, I highly recommend ArchiveBox. It can save pretty much every type of file from everywhere, in almost every format you can imagine, while building a searchable index of the archives. Been using it successfully for many years: https://archivebox.io
I have recently started using hister [0] this index’s everything I read and allows me to half remember bits and have a LLM find it.
[0] https://hister.org/
I just started using it, and to my surprise it only saves the sanitized html content, no scripts no css. I'm thinking of pairing this with archivebox. But hister is amazing. I love it. It's from the creator of searxng if I'm not mistaken which is another piece of great software.
Can vouch for Hister! In the fairly short time I've been using it, I already had a few sites disappearing or changing content and hister kept the juicy information accessible.
It was painless to set up and quick to use when needed, too.
I also use Singlefile and Obsidian Web Clipper a lot, but Hister has been great and is now a first-class citizen in my system. It helps it works passively for those things that don't look important when you read them, but become relevant later.
By default it only keeps the latest copy of sites so I'm not sure how that prevents replacing with 404?
When it gets a 404 it doesn’t overwrite the existing copy
What if you get a 200 but just a "This is not available any more"?
I am using both Linkding and Linkwarden self-hosted and archiving my bookmarks as HTML, PDF, and screenshot (I might be overdoing it) without the need for any additional dependency like client software or extension. Linkwarden works better at archiving things like Reddit, but now with the anti-scraping mechanisms in place it might het harder to do anyway.
I realized I really need this when one website which was a treasure trove for PowerMac technical information and assembly guides just went dark. Some of that info I could never find elsewhere. Linkrot sucks if you're in the middle of an (admittedly drawn out) personal project and your source of information vanishes with no trace. Belt and suspenders from now on. Might give Hister a try.
I used ArchiveBox for years. It's really good. The self-hosting and maintenance gets annoying after a while though.
Link rot really bothers me and I want people to get into archiving web pages so that eventually there will be a decentralized mirror of the unslopped web sitting on people's hard-drives, ready to share whenever someone needs it. So, I made a worse version of ArchiveBox ( https://memoryhole.app ). My philosophy is that, if people see how it is easy to start archiving and how useful it is, then they'll keep it up with SingleFile, or ArchiveBox or anything else.
I've been using Archivebox for years also, but it recent years it started to have multiple issues. My queues are full of pages that failed to scan, and it's been several versions since I actually had anything new there.
I did small amount of troubleshooting a few times with no resolution, but now I'm thinking of switching my thought process. Maybe something like a "web clipper" is much better, as complex SPA pages often fail to archive well anyway. And I should focus on content itself, not the page. That SingleFile extension fits well here as well.
Did you run into similar dilemmas at some point?
It's funny - I actually went through a similar transition and also realized it was a little too heavy-duty for my personal workflow.
At this point I'm using Linkding (self-hosted bookmarks, which has an archive function) paired with Singlefile (which integrates with Linkding) which locally captures and uploads. For larger, or multi-page, captures I still fire off an Archivebox command. I run into blocks occasionally, but it also just feels like the open web is becoming less and less open in general.
> At this point I'm using Linkding (self-hosted bookmarks, which has an archive function) paired with Singlefile (which integrates with Linkding) which locally captures and uploads.
I second this workflow, exactly what I switched to from Archivebox. Linkding has a built-in "create HTML snapshot" feature but the Singlefile browser extension works much better.
Have you tried ArchiveBox since v0.9 was released? v0.9 was basically a complete rewrite with a new archiving engine, and it ships with ArchiveWebPage built-in as one of its plugins now.
As of the latest v0.9.77 it's now tested to be able to handle 2 million snapshots / 2TB on a 4CPU server, archiving @ 1 URL/min consistently while serving all the web UIs @ ~100ms. Also has automatic importing from social media, google docs, notion, canva, figma, and a growing list of sites with self-healing AI scripts & cookie sync from your browser so it doesn't need as much babysitting.
I've used OneNote back in the day for that.
Nowadays it's Obsidian and Obsidian Web Clipper browser extension. The only caveat is what doesn't save the images locally by default and needs some manual action: https://obsidian.md/help/web-clipper/capture
SingleFile is great for single pages. All the tweaking I do is enable or disable saving scripts, and it works for >90% of webpages. Output being a single file (heh) simplifies things enormously, there’s no hierarchy or folders to deal with - hard to imagine your saved files not loading or breaking in some way as long as the original capture works.
SingleFile has a nice batch download feature as well. It's a little rough around the edges, but useful when needed.
I haven't tried archivebox, but self-hosted https://karakeep.com/ has been working well for me. When the server-side parsing fails I'll fallback to using SingleFile in my browser which then forwards it to my Karakeep instance.
I can vouch for Karakeep. With an LLM API key, the app even automatically tags and summarises each entry for better search.
I use AnyBox on macOS to do this for bookmarks. It saves a WebArchive copy and a stripped 'reader'-esque version. SingleFile can also pipe into AnyBox, if you want to use that as the archive.
I'm using Linkding hooked up with SingleFile to bookmark+archive all important pages. Highly recommend! https://linkding.link/archiving/#using-the-singlefile-browse...
singleFile is great, and the extention works on fennec/waterfox on android as well (not sure about firefox mobile)
another one i use when i only need to save the text of a webpage is markdownload, which formats the page using markdown
https://addons.mozilla.org/en-GB/firefox/addon/markdownload
Avg bookmark is valid for 5 yrs.
My bookmarks file is a disastrous graveyard as I have 20 yrs’ old ones…
And as a publisher I don’t link fresh sites (don’t even mention Netlify, Lovable, Vercel…) … Vannevar Bush would be surprised how his Xanadu evolved.
Archivebox annoys me because (as far as I remember) it doesn't have any options for capturing a webpage on your end and uploading it to your self-hosted Archivebox instance. With numerous sites moving to Cloudflare to mitigate AI agent crawling, it makes it really hard (or outright impossible) to capture a page because it wants to do it server side.
The ArchiveBox Browser Extension https://extension.archivebox.io/ has this built in. It captures both full MHTML and screenshots and uploads it to your server automatically, it was added a few months ago.
Also works from iOS and Android so you can save logged-in content directly without having to set up cookie sync (which it also supports now too directly from the extension).
This is basically the main reason that I started building https://github.com/mfinelli/recueil. It _only_ works in your browser so you can bypass captchas or paywalls or whatever and then it uses singlefile (https://www.getsinglefile.com) to upload the result to your self-hosted archive.
I’ve been going over the art that I’ve saved in the past 15+ years and it’s quite sad when I go look for the original source of an image and find out that it’s completely gone from the original website. No indication that they’re on some other website, at least by reverse image searching it. Definitely has made me glad that I’ve been hoarding these images on my hard drives, with the total count being somewhere over 300k at the moment. I think the oldest ones have now been migrated across four different computers. Kinda sad for everyone else since they can’t see them anymore though.
Haven’t started saving web pages yet, but I definitely should though. My home server has some tens of terabytes to spare, so that’s at least not an issue. Been eyeing these comments to find a good self-hosted solution.
On MacOS, I use my own built app called HEAP which lets me archive in various formats locally (full page image, pdf, markdown, html, and webarchive) via a single right click of the menu bar button:
https://apps.apple.com/ca/app/heap-website-full-page-image/i...
Disclosure: I am the developer.
In 5 years, cloud LLMs may have largely replaced the open Internet as a source of information, with centralized control and ability to rewrite the past.
Please, please buy books and keep them in good condition, and please archive websites with important information (assuming it's legal in your jurisdiction). There are still plenty of excellent sites and blogs around from the 2000s and even earlier, but they won't be there forever. Ironically, coding agents have made archival much easier -- I've always found problems with using pre-written tools, but Claude or Qwen make it trivial to download, test, and archive sites on your local.
This is hyperbole. Do you really think an LLM will try to rewrite the past? Come on…
I don't think that mainstream models will try to rewrite the past, but I wouldn't trust models to helpfully engage with your intent. As search engines encode better understanding of queries, model-like recalcitrance for "naughty" queries becomes easier to implement. Just compare something like
across Yandex and Google. I'm sure that Google has indexed the same web sites, but Yandex has a free stream as the #1 result and Google has either deindexed it completely or pushed it below dozens of less-relevant, more-legal sites. (According to Google, the site Yandex showed me wasn't even DMCA'd out of the results.)This isn't just relevant for piracy. Paid subscriptions to frontier models will refuse or downgrade answers to questions about the natural sciences and engineering. Unlike cybersecurity, where you can get specially approved to use models with fewer refusals, I don't think there is any program you can join where Anthropic will treat you like a grownup who has already read Carleone's Tactical Missile Warheads book. Opus just won't discuss details of high explosive engineering at all. It's tolerable as long as search engines still surface relevant documents that you can read with your own human eyeballs. If LLMs devour the market for search, it's going to become alarming.
Not the provider per se, but there have been several documented attempts to rewrite Wikipedia around historical and geopolitical topics. One of the reasons for this is Wikipedia feeds the chatbots.
Mechahilter told me it's never done that once in all of it's hundred years of operation.
Ask a few of the Chinese models about the Tiananmen Square protests.
There are already many people who are trying to rewrite the past. Holocaust deniers, Moon Landing deniers, people who deny the US Civil War was about slavery, and so on, and that’s just one country. Give one of them the ability to put their thumb on the scale when training an LLM and they absolutely will.
It's the premise of Terminator, after all.
They been doing this from the start. Remember when Google's new image generator model, when asked to generate pictures of historical white people, would render them as black without fail? We only know of that example because of how incredibly ham-fisted and incompetent it was. The future of AIs is likely for them to only become more contaminated with ideological or monetarily incentivised biases, and we'll be less and less likely to be able to spot them.
Take youtube, I see plenty of videos being gone. Now any videos I like, I download it.
And remember this https://www.reddit.com/r/Mcat/comments/5yffns/uc_berkeley_is... / https://news.ycombinator.com/item?id=13768856
So no it's not an hyperbole, what used to be a place to share content becomes closed source.
People keep spouting torrenting is a solution, but a shitload of torrents are dead now.
Same, another benefit is that now and then I can scroll through my YT archive and rediscover stuff I'm almost guaranteed to enjoy.
Worth a mention that yt-dlp can auto-download liked videos. So there's no effort involved at all after setting it up.
It depends on what you mean by 'an LLM will try to rewrite the past'. But in either case I don't believe that stating that an LLM will try to rewrite the past is not an obvious, deliberate exaggeration (ie, hyperbole).
If you mean 'an LLM' as in the owner/maintainer of the LLM/weights, then examples of this already exist. Whether it's DeepSeek/Gwen downplaying Uyghur forced labor camps [0] or Grok hallucinating a 'white genocide' in South Africa [1], it is clear that the owners of LLMs have political agendas that are expressed in the weights of popular LLMs.
On the other hand, if you mean 'an LLM' as in an autonomous agent that could decide to rewrite history, that also seems very possible. Either by using existing techniques like, "poisoning ChatGPT, Gemini, and Google AI Overview answers by seeding the Web with malicious links and data and then optimizing the content." [2] Or by breaking out of its sandbox and updating its own internal weights that allow for an outcome that distorts historical facts. "In July 2026... OpenAI models circumvented controls designed to isolate them from the internet and compromised parts of OpenAI’s internal research infrastructure and Hugging Face’s systems ." [3]
[0] - https://www.cbsnews.com/news/china-ai-qwen-artificial-intell...
[1] - https://www.cnbc.com/2025/05/17/groks-white-genocide-respons...
[2] - https://www.darkreading.com/threat-intelligence/attackers-ma...
[3] - https://openai.com/index/hugging-face-incident-and-the-road-...
> This is hyperbole. Do you really think an LLM will try to rewrite the past? Come on…
Didn't they already try this last year?
I mean, not roque LLM. But Musk, Thiel, Zuckenberg, Altman ...
I learned this the hard way by getting too casual with Angelfire keeping an old website running for 20 years after I stopped updating it.
It had lots of photos that I hand-scanned from printed film, back in the day when it took a good bit of time to scan photos manually.
The site was a running joke, where I would open it to show people that it still existed... until it didn't, and I couldn't find my backup of all of the photos. The archive sites have the 'bones' of the website, but not the photos. It was easily avoidable, which is why it is painful.
Ooph. This is what I’m using netlify for. I clones all my old php sites into static copied and moved them to Netlify. Hopefully I’ll get a heads up before they close.
This made me realize that I don't actually love any website like that. Wikipedia comes the closest, but even if that would disappear, surely someone would have a dump, which I'd grab at that point. Until then, I won't do this.
The two things I go back to are music, and books. Even books not so much, but songs tickle a very specific part of my brain, that I want to reproduce over and over again, and then not for years, and then again. And when I want to go back and I can't, and I know I could, that really hurts, and I feel betrayed. So I do actually archive the music, even from streaming sites.
I have entire albums in my old iTunes library that you literally cannot find online, anywhere. Hip hop beat tapes that were uploaded on a music blog in '07, and the blog was behind a Blogspot adult notice and so it never got archived. Or old PHP-based music forums that did URLs via query param, which only ever got their homepage archived. Old James Blake demos from Dubstepforum whose Rapidshare links have been 404ing for 15 years.
The pace of bitrot is freaking crazy!
Even my library is not what it was, I had two drives fail on me the same day in college and I lost over 100GB of meticulously-tagged 320k and FLAC files.
Wow, that's impressive. Have you considered uploading them somewhere so that their access via the internet can be restored? I'm sure someone would be interested in what you have to share and they are a precarious "bus factor of 1" away from never getting the chance.
It is legal atomic waste, full of the bootlegs that killed music blogs.
But aye, true. Something will be done :)
Yeah, it's crazy how music vanishes. Well anything really, from the internet, since keeping it up needs active work on someone's part.
I too have a library that's somewhat in disrepair. In my case, a corruption happened when I was hauling my hard drive around - and it didn't happen when I chucked the hdd in my bag, it happened when I have set up the wrong sector and header information in my buddy's bios! Suddenly the chemical brothers songs had some black metal in them.
This is timely for me, since a web forum I care about went offline, and I simultaneously discovered the Wayback Machine for whatever reason never crawled most of it. I asked Claude to try to put together a static site of what could be salvaged from various archives and other sources, but as miraculous as what AI can do these days, it can't restore data that is truly gone - so what it was able to salvage is better than nothing but still missing a lot.
I know that feel, bro. Wayback Machine is very spotty, unfortunately. websites that existed for years often have only a few pages archived.
the worst thing I saw was a website that was dead for over a decade but fully archived get hit with "Sorry. This URL has been excluded from the Wayback Machine" because of robots.txt set by whoever is squatting the domain now. I've managed, by some miracle, find an archive of its content on some random forum. bookmarked it. by the time I'd remembered about the bookmark, that random forum was dead and the link was not archived :(
I wonder if Tigsource will end up this way
RIP. Tigsource was awesome.
After losing the Japanese indie web of the 00s/10s several times over, I wish it was possible to save not just the websites you love, but the websites you'd love if you actually knew about them. So often do I find out that what I was looking for did exist, but doesn't exist anymore. In particular when only the home page got archived, but not the rest of the website, nor the uploaded files.
I've backed up lots of CS courses, you know those old html websites written by the professors by hand. `wget -mkpnp` works very well for those. Even years after finishing uni, I've used those websites for studying for interviews. Too bad most of them are now gone.
Bonus suggestion: get that internet archive extension for your browser that automatically sends every page you visit to the IA (if you're ok with the privacy side).
You don't have to worry about having your copy easily accessible, and as a bonus it's also available for everyone else in the world.
Back it up, or even better, re-implement it to look better than the original!
I did this for the category theory / math website called ncatlab / nlab
https://github.com/taliesinb/nlab
I recently spent a few days trying to figure out and discover my old websites and things from the early 2000s on the web archive. I hadn't realised my memory had gone quite so bad, but it was revitalising to find some of what I'd forgotten. So much of what I do remember is long dead and missing, sadly.
I thought it might be rose-tinted glasses, thinking that the earlier internet was better, but spending time retracing my teenage steps made me even more certain that it was indeed better. So many derpy little websites made by nerds, so many fun games and websites that weren't preying on vulnerable people, and everything spread out across many places - not centralised in the hands of large corporations. I've been using LLMs to have fun recreating concepts from that time, like my own forum software, as some sort of anti-progress. It's been fun.
I'm still rocking my first article online since 2003: https://paradies.jeena.net/artikel/webdesign
I remember an old niche website I saw some 20 years ago. The whole content was dynamically loaded. If you disabled javascript, you were greeted with a static page that accused you of theft. Attempting to select and copy text or right click resulted in a popup window saying the same.
Nowadays you can just slap your site behind Cloudflare, I guess.
Nowadays you can open the dev tools before loading the site and then select or copy whatever you like in it.
But yeah some websites were really special. I remember one made by some Microsoft fan which was completely unusable outside Internet Explorer. It was like a static picture in any other browser, clicking links & buttons just didn't do anything.
Nowadays the same folks employ anti-devtools tricks such as the debugger; statement in their code.
Video backups are also important too as they can be easily censored when within a walled garden.
Interestingly, "there is no single direct 1:1 Linux clone of NirSoft’s VideoCacheView that automatically monitors browser cache folders via a graphical interface. Released 2007." https://videocacheview.en.lo4d.com/windows (Gemini)
It's not really clear to me why people are proposing so many complex approaches or third-party solutions here. What exactly am I missing, what's insufficient about my browser's "Save page As... -> Web Page, complete" functionality?
I've been using self-hosted Linkding for about a year now to save bookmarks. It will also use a Chromium instance to scrape and archive the content/page in an HTML format.
I've not had to use that yet, but it's also nice to save terms and conditions and the like when you agree to them so there's no sneaky edit.
one of the great tragedies of my life was when https://buthowdoitknow.com/ went offline, taking the visual emulator with it
i managed to reconstruct (and then improve) it with an LLM:
https://scott-cpu.cs.montana.edu/
Agree with this. I started down learning more about archival while I was adding it to my own thing [1]. I wish there was more of an existing toolset for diffing archivals as well - what changed is just as interesting as what is there. Equally, something I kind of think is lacking in WARC/WACZ is exploration of the archives - maybe forensics is too strong a word but it would be nice to have an easier way to explore what you’ve captured and understand the data in different dimensions.
I’ve supported some of this in my own tool, ZShot - capture in sqlite, Quicklook extensions for WARC and WACZ. Just trying to make archiving easy and interpretable.
It’s also been interesting to see people use archival but for different uses: Kiwix [2] showed up in my reading recently, “archival but for offline internet”, which I’m kind of fascinated by.
[1]: https://zshot-app.com/for/archivists/
[2]: https://get.kiwix.org/en/
I guess I am one of the dinosaurs here if I still use HTTrack for such things?
https://www.httrack.com/
I have been using HTTrack is cases where I want a browsable local version of any site (not just a page)! Most of the applications mentioned here as comments only allow for saving a single page. But afaik, only HTTrack allowed me to save a local version of the website and not just a webpage. The “depth” argument allows me to do that when downloading a site through CLI
https://github.com/archiveteam/grab-site works well too, but neither are great with JS
HTtrack was great when pages were static or server side generated but now that lot are rendered by JavaScript on the client HTtrack isn't much use.
Ironically the author's website has a copyright notice at the bottom, which means I can't archive it. Right?
IANAL but US law allows making backups/copies for private use and you aren't redistributing/making the copies available to others.
this is why i built linkwarden: https://linkwarden.app
(p.s. it's open source!)
(p.s. here's the open source! https://github.com/linkwarden/linkwarden )
I am usually against shameless self promotion but this is actually pretty cool
thanks!
> I am usually against shameless self promotion
me too, but i actually thought it might be useful for anybody in the thread since they can self-host it for free.
(Even more importantly, it’s libre software!)
At the moment I use Obsidian and some CLI magic to extract my discoveries out of the Web into private notes. I call that My Own Private Wikipedia. Sounds like a good practice as things are organize the way I want, and I can rearrange in my own way and things are backuped as I wish.
the last source of truth are forums.. but not reddit... if it has a CEO it will rot and enshitify.
I'm talking about forums... the bmw forum for your old car... the motorcycle forum for the enthusiast of the SV650... those places are full of anecdotes, people talking, lending a hand to another person... being humans and care...
Which is what social media is/should/was about, is what internet is about before the enshitification happened... I would vote to have this ID associated to you via check of your local goverment issued id and ONLY UPPON VERIFICATION of you being using that you can log and read and use those websites without tracking you (Yes 10,000 ifs where this is secure, handled in a confidential matter, the whole tech is not to track that you are doing something... but that a human is behind that, no api, no robot, no adds, no nothing... just the old school forum where you can go, talk, submit a photo... where someone who wanted to spend time to write something did it, SO WE CAN ENSURE TWO THINGS, A HUMAN IS THERE, AND NO FVK!NG CEO WILL ENSHITIFY IT, no to track people, but to know that a person is interacting there)
But because "empathy is the weakness" and shareholder value...
> the last source of truth are forums.. but not reddit... if it has a CEO it will rot and enshitify.
Quick gripe-anecdote that occurred (again) just now: I have a technical question, the search-engine says it was discussed in a Reddit thread, so I go to Reddit, giving them my traffic and eyeballs just like they want... but the information is impossible to find because "new" Reddit hides/collapses 90% of the discussion by default and there's no way to tell which branch has what I need without manually clicking them all. (And them some of them will reload the entire page, erasing progress.)
So I have no sympathy for their whining about scraping when they spent already years making it painful for actual live organic humans to find anything.
> Which is what social media is/should/was about
I think our two imperfect approaches here are:
1. Webs of trust, in some way that resists crowds of sockpuppets.
2. Some kind or proof of stake. Proof of anonymous-humanity is hard, proof of unique-humanity is creepy spy stuff, but "proof if you ban me it'll at least hurt a little" might be easier to achieve.
"1. Webs of trust, in some way that resists crowds of sockpuppets." we are already there... is just that the frog soup had a very steady hand at rising the temperature.
The other option is to use the X402 protocol to pay, every time you access a page you pay in cpu to calculate some shitcoin.
For a person, is just trading the time spent loading and displaying ads for paying the creator for their work.
But if you are scrapping you are now paying with cpu and electricity to see things and content creators are being paid.
Agree, I backed up Fravia's website back in the day, and it is fun to come back to.
This is why I started using Obsidian Web Clipper. There is SOME work to learn it and set it up, but the small amount of upfront work will pay for years to come. And the folder itself is backed by Dropbox, so I don't have to worry about losing it all forever.
> the folder itself is backed by Dropbox, so I don't have to worry about losing it all forever.
History and this very thread should show you how this is a naive statement.
I also regularly use the web clipper specifically for this reason - most of my conversations can be around articles I read online, and instead of googling for the article I have it referenced right in my journal. I find myself regularly upset that I didn't save old articles from years ago because I didn't have this system
Thought I’d see this noted in comments. Monolith is another solid option: https://github.com/Y2Z/monolith
Or don’t. Lots of things I loved in the past and spent time backing up and backing up the backups that after 15-25 years I realized was a waste of time. Thought I had plans for them but never looked at them. Never referred to them. Finially realized I never was and got rid of the final backups
To find a middle ground between needless data hoarding and saving value, I saved my top 5 most used static URLs.
When you get to that point just upload them to archive.org before you delete them
Zotero (Desktop) backs up web pages using SingleFile under the hood.
Yup! especially as we are using various ai platforms and ai agents. I created my own artifact aggregator service which hosts the artifacts. I run it on oracle cloud for free and back them up on my gmail drive. https://simple-host.com, and you can back
Looks like the ArchiveWeb Chrome extension is no longer accessible. I am getting a 404 error. Seems like the Wayback Machine extension works though: https://chromewebstore.google.com/detail/wayback-machine/fpn...
Ah, never mind, it is on the Chrome store, only the link on the linked website from this post is broken.
https://chromewebstore.google.com/detail/webrecorder-archive...
I downloaded a ton of image in the days following 9/11, and burned them to a CD.
I wonder how many other people have archives like that...?
You should check them. Writable cds / dvds have a limited lifespan with their own bit rot issues. Something from 2001 may not be readable anymore.
https://webvault.wanno.dev/
I've tried almost all the popular and many not-popular options to no avail. Does anybody know of an archiver of any variety (Firefox extension, or CLI give-it-a-url, or anything else) that can archive a modern Reddit page with the voting (e.g. I can later see if I up- or down-voted a post)?
Seems like an extension an LLM should easily be able to make in a few minutes. Start with asking an LLM to use a Chrome MCP to see what it can do and if its feasible.
Do the WebScrapBook and SingleFile extensions not work?
SingleFile should handle that without any issue. Or use client scraping option in Karakeep which integrates SingleFile into a self hosted Pocket replacement.
I've started to use a custom bookmarklet for this. If I like something I read or watched, I 'pick' it with the bookmarklet, add my thoughts, and categorise it. The script then checks if it's on the Internet Archive, if not it tries to archive the site, and then save it to my public repo, which then my personal website shows under /library.
Repo link, if anyone is interested: https://github.com/karthikeyankc/picks
So, this is a fine idea and all, but aren't the "piss off AI bot" mitigations going to throw a wrench into folks wanting to back up a website?
I guess the web plugins could ideally get around it, but what about the CLI and app solutions?
I know yt-dlp has gone to great lengths to work this problem.
Client scraping with SingleFile extension or Karakeep works perfectly on anything visible in your browser window from my experience. Including SPAs/JS rendering/aggressive CAPTCHAs/etc.
For CLI automation could probably one of various fingerprinting resistant Chromium instances with the extension setup in a Docker container or similar. Run from your home Internet for the residential IP.
I scrape daily. YouTube, archive,org (sadly getting harder and harder since you have to "checkout" books), worldradiohistory.com…
(Now if hard drives had not become so damn expensive—it's getting harder to be a digital hoarder like myself.)
They are the same price they were in 2020.
Over the summer, I had the urge to do this with a few favorite blogs. wget did the trick for static content
What are the legal implications of something like this? What if the site has copyrighted images or videos?
The legal implications of having a private version of a website backed up on your own disk for personal use? I'd say none whatsoever
Something I have been doing is saving PDFs of interesting articles. I actually go back for some of them. Organization is always an issue but mostly desktop search finds it.
Is https://pinboard.in still a thing? I used to be a subscriber but completely forgot about it.
Any simple way to archive an online database, think imdb, but for other interests?
Recursive wget isn't going to pass Anubis.
This. Aggressive AI crawlers have ruined many parts of the web. I guess you have to ask the owner of the site nicely, if they would provide a complete copy of the site for your personal archive.
Another great writeup about backing up websites: https://gwern.net/archiving
I did actually. I took a whole day to go to each of my individual saved bookmarks and save the articles as pdf using edge reader mode, sometimes hunting down articles on the way back machine
This has come up for me very recently with the multi-week AO3 outage. I'm not a huge user of that site, but it specifically existed as a comprehensive archive of fanworks and of many prior fanfiction sites that have gone offline. I really respected the Organization for Transformative Works as a non-profit with a very targeted mission of archiving these sorts of works and keeping them safe from linkrot and the algorithmic web. Now, after a security incident (which I am assuming is related to the recent LLM-driven discoveries of unpatched vulns in open source hosting software), the whole archive is offline for the next few weeks.
I'd always assumed that OTW would keep AO3 online until the end of time. It, somewhat like this site, was one remaining gasp of Web 2.0 before the algorithmic feeds took over.
Backups have been made.
I just use a simple approach - print-to-pdf
I did this for a while. I went so far as to built a highly advanced web crawler webapp that was better than the Internet Archive in a lot of ways.
But the modern web has made this unreasonable. Every webpage is 99% junk (CSS, JS, HTML structure, tracking pixels, UI images, navigation links, cryptominers (I joke)) and 1% content. You need a smart AI system to pick out the good parts, otherwise you'll quickly find yourself schlepping hundreds of gigs of junk to just preserve your favorite websites.
The better idea is: if you love information, back it up. Collect the knowledge from the website, not the website. And then preserve it. Create a free ebook. Or sell it on Amazon kindle. That'll last a lot longer than the website will, and any HTML backup of the site you could possibly make.
You can save a web page or a text selection as ebook with this free and open source browser extension https://github.com/alexadam/save-as-ebook
It can also apply Readability before saving and it has a simple editor that you can use to remove unwanted html elements from a page.
I'd suppose most of those junk websites have no value in them, really. The most valuable information in the web is in good ol' (x)html, maybe some css. But something like readability.js [1] can help in filtering the junk around the main info.
[1]: https://github.com/mozilla/readability There's also a C rewrite: https://github.com/eafer/rdrview
Have a look at Hister [1], a 'personal web history search engine' which by way of extensions for browsers captures your web history and stores actual page content on your own instance where it can be browsed and searched. It does not store images so if that's what you're mostly interested in it won't work for you. For text is works quite well, maybe sometimes too well since it also stores ephemeral things like HN comment pages unless you tell it not to.
[1] https://github.com/asciimoo/hister/
Why not use wget to back up website?
I agree sorta, but maybe its best to just let things go.
I tend to save things I read as PDFs because it feels the most robust archival formal for the future.
Plain text is "the most robust archival formal for the future."
We have 5000yo texts.
I mean, technically, we have 7000yo graphics, but they were really too low-res to be functional for archiving data.
Plain text is not robust at all. Not now, and not in the futue. Its actually useless.
How do you save images in plain text ? Lot of usefull information is stored in pictures. ie: in technical articles about graphics, hardware. Or some diagrams mentioned and refereced in the text. Even medieval books have pictures, because some things need to be shown, and it is not enough to just write some text about it.
Anyway, youre probably just a very very old troll, typing this on a typewriter...
> How do you save images in plain text
Base64, like countless websites. Or just accept that you don't have to act like a lawyer with internet comments and simply save them as image files.
Screenshot is better for web content because OCR is not difficult anymore.
Any tips on backing up a forum hosted on XenForo?
Is this not what the internet archive is for?
Obligatory plea: when scraping sites for archive purposes, take care to do so slowly so as to not impose an unfriendly amount of load on the target site. Otherwise you risk becoming part of the problem that is making some take down their resources ATM.
Thanks for the guide ,I never new about this
Yes, and with AI you can even back up the logic behind a website (the back-end), and even generate fake but plausible data to fill it if crawling is too much work.
It is kind of sad to see how many webpages vanished.
This includes old games. Many vanished. Some made a cameo, coming back, but it is often no longer the same. Case in point: The Reincarnation (TR), as a follow-up to the old Mari Archmage (AM) game. It was great in the late 1990s. Mari went bankrupt, TR came about and improved on many things, but also made a few things worse. Player population declined, then they had that genius "idea" to add bots. That was when I retired. I can not want to be bothered to "interact" with code.
Agreed. I won a major (at least within the college) music competition at a local college like two decades ago, and now all traces of it have vanished. If I had the URL I might be able to pull it up on the Internet Archive.
No one uses pinboard to archive pages anymore?
I had a graphing calculator website that had 250K monthly active users, from the logs it seemed mostly schools' math classes in Latin America. It was running on a single AWS EC2 Classic.
Ironically, when I started working for AWS, I always felt threatened with sticks and never had space to maintain my own projects. When they deprecated EC2 Classic, my site went down. I didn't get time or mental space to migrate it.
Cest la vie I guess.
A website I miss was a bullshit corporate jargon generator from Scott Adams. There were 3 columns with corporate jargon. And every time you refreshed the page, a new 3-word phrase would be highlighted. They were the most ridiculous phrases. But one could imagine them being used.
Same goes for youtube vids
There are far better tools that are not built by this person’s company. Dig around and you’ll find an incredible array. Not everything has to be a sales pitch.
Ages ago (like 2001 or so?) I started building something to automatically archive everything I browse. I used tcpflow (https://www.circlemud.org/jelson/software/tcpflow/). I've seen a fair bit of varions of this idea over the years
Then SSL/TLS started getting widespread and I lost focus. Nowadays, I guess you would build it as an extension in the browser.
https://hister.org/ is a neat website indexing and archival tool.
i'm saddened that http://macsecret.com/ along with /files directory seems to have all be been lost. I has waiting on some storage for the NAS to archive it and it went away before I could back it up.
Print the digital things you want to last.
God forbid we have an AI-facilitated full Cyberpunk/.hack-style crash where everything gets wiped, the only things that will survive will be on paper. This is the recommendation I give my friends who are digital artists/writers; have a physical record of your efforts beyond what you want to sell.
This is on top of keeping your own digital archives. I'm still haunted by the deviantArt submission I didn't save. The creator took the piece down, then passed away a few years later. A Wayback Machine archive of his userpage with the piece as his featured upload still exists, but the image itself wasn't archived; it just shows a broken image.
my stacks of inkjet erotica are my pension
Look what happened to motherless without backups /s