rsync 2 hours ago

Free account, indefinitely, to whoever needs it to store this data … including the storage vendor.

Just email…

arjie 15 minutes ago

It's funny to me that entire companies exist that have a fraction of storage that I personally have in a datacenter on flash let alone in total. 50 TiB is so little. It's like $7k annually on R2 and under a thousand bucks on S3 Glacier Deep Archive. I suppose it's just whoever knew the cousin of whoever at whatever position of authority but it's funny how just using deep glacier means he would have such insane margins over paying Iron Mountain.

Crosseye_Jack 3 hours ago

Ok, it sucks, losing data always sucks, but what ever happened to the 3-2-1 backup rule? even if your off site backup provider goes tits up, your on site backup is still there.

If you don't want on site, a second off site provider, 50TB on backblaze would be an extra ~$350 per month (prob less if they were to commit to capacity instead of using their PAYG service)

  • embedding-shape an hour ago

    > but what ever happened to the 3-2-1 backup rule?

    I wasn't 100% sure, but the data backs this up[0]; the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.

    - [0] Interestingly, Ngram (https://books.google.com/ngrams/graph?content=%5B3-2-1+backu...) says ~2010 while Trends (https://trends.google.com/explore?q=3-2-1%2520backup%2520rul...) says slow build up from ~2020, guess "everyday" language would sit somewhere around those two different truths.

    • layer8 18 minutes ago

      The concept of multiple copies in multiple locations already existed when backups on CD-R were common, so 20+ years ago. The “2” in the 3-2-1 rule refers to using different physical storage media and predates cloud storage.

    • bityard an hour ago

      I can pretty clearly remember hearing about the "3-2-1 backup rule" by name 10 or 15 years ago on the Security Now podcast with Steve Gibson. So it is _at least_ that old, but probably even older. The concept of having multiple backups is not at all new, however. Since I've been involved with computers (~30 years), I have heard folks say things like "two is one, one is none" in reference to backups and data replication in general.

      Also, this TV station has not been sending their show archives to Iron Mountain for the past 70 years. Probably what happened was: In 2019, they had an on-site or nearby archive. Then some enterprising IT director came along and convinced management that they would save a bunch of money and time by outsourcing everything to a third party who said they could do it for cheap. Very confident in their genius, they then scrapped their own archive entirely.

      Part of me pities them for ending up in this situation, the other part of me believes they got everything they asked for.

    • BeetleB an hour ago

      > the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.

      They signed up with them in 2019. Not long ago.

      Even if it is a "new" concept, common sense isn't. Pretty much anyone who cares about their data will do the due diligence, and this is not something that requires technical chops. This has been a concern for well over 20 years!

      It really sounds like some senior management trying to be cheap.

    • olyjohn an hour ago

      This has been a rule for like 30+ years (I have only been in the industry that long). Whether it was called "3-2-1" before that or not, the practice is the same. Have an onsite backup, have an offsite backup. This isn't a new concept at all.

    • Crosseye_Jack an hour ago

      The 3-2-1 rule is ~20years old at this point. But just because an archive is "old" doesn't mean we can't apply modern standards to that archive as time marches on, otherwise the BM_ARC would have never been build and opened a couple of years ago.

      I mean, that 70+ year old archive must have been digitised at some point, unless Iron Mountain was storing the reel to reel tapes and/or film from the original recordings. So some "modernation" has been applied to it since its creation.

  • lenerdenator an hour ago

    It's worth noting that PBS affiliates are non-profits that have fairly constrained budgets, especially now that the Corporation for Public Broadcasting has been dissolved due to DOGE.

    Non-profit IT has always seemed, to me at least, to be a land of compromises. There's rarely enough money, knowledge, or time to do things "the right way", so you have to do the best you can. If I could only choose one backup to have, it'd be remote on a provider like Iron Mountain. It would seem as if they are storing things on Iron Mountain through an intermediary company, who may or may not have actually paid the bill.

    So, y'know, fraud's a possibility here.

    • Mistletoe 26 minutes ago

      The guillotine is too nice for the people that dissolved the CPB. That just makes me so mad. How many children learned via the CPB like me? America isn’t even the same without CPB and I guess that’s why these traitors did it.

ApolloFortyNine 36 minutes ago

>storing over 50TB of data

I thought this would be like the backups I've heard broadcasters have of every broadcast they've ever had (I've heard of people being able to request 20 year old news clips before).

But just 50TB of data? Duplicating that would have been cheap and trivial.

  • thekid314 13 minutes ago

    Yeah, we are talking about 3 20TB drives in a closet here.

    I'm an independent photographer and I have my 20TB drives behind my monitor, backing up to backblaze with an offsite synology. Keeping this archive up does take time and money, but PBS should have the budget and IT deparment for this.

thelastgallon 4 hours ago

This might be the website: https://www.osstorage.com/

I wonder how they got the contract? Seems to be 4 people 'employed' at this company. Of which, one is retired, one is "Linkedin Member", one is a controller and the last one is a social media person: https://www.linkedin.com/in/isabelle-anderson-7aa250b5/

So, just the person to deposit checks and a social media person to get paid. Who exactly is left to work on storage infrastructure?

https://www.linkedin.com/company/osstorage/people/

  • ndiddy 3 hours ago

    From the article it seems like Iron Mountain was doing the actual storage, and OSS was just acting as a middleman and taking some money off the top.

    • chuckadams 3 hours ago

      And they sent OSS a legal demand letter to keep paying the Iron Mountain bill for their data. If OSS actually was using IM and not just running a scam, then the data should be pretty safe, if just not accessible for a few weeks/months.

      • pavel_lishin 2 hours ago

        I guess that depends on how long Iron Mountain has to keep their data once OSS stops paying - and we don't know how long ago OSS's money ran out.

    • bityard an hour ago

      To be somewhat fair, Iron Mountain is just a storage facility. OSS were likely the consultants who worked out how to get the archival data from the station's tapes into Iron Mountain and then get the data back out when needed. So they probably offered a worthwhile service, they just eventually stopped doing it and left the station in a lurch.

treetalker 4 hours ago

This underscores the importance of the Internet Archive and people like Marion Stokes (https://en.wikipedia.org/wiki/Marion_Stokes).

  • ChrisRR 3 hours ago

    Don't trust Internet Archive as your only backup method though. There has been instances where they delete files without warning with the only explanation being that they don't meet TOS

    • embedding-shape 2 hours ago

      > Don't trust Internet Archive as your only backup method though.

      Agreed, no doubt about it.

      > There has been instances where they delete files without warning with the only explanation being that they don't meet TOS

      What sort of content? All the content I've judged to be of public interest and uploaded to Internet Archive, is still online there today. But I'd understand that depending on the type of content, that might not be true for everythibng.

    • toomuchtodo 3 hours ago

      Inability to access items does not mean deleted. In the case of PBS content, it is unlikely access would be lost. I cannot speak to your use case, but the Patron Help Desk can confirm if an item is relevant to their preservation efforts if you'd like to check before uploading. Email info at archive dot org to inquire.

      Someone dumping their Twitch stream or random Youtube content into the storage system will certainly have that content deleted and be banned if prior arrangements have not been made, for example. Don’t do that.

      https://old.reddit.com/r/DataHoarder/comments/sq6wbq/please_...

  • toomuchtodo 4 hours ago

    It's incredibly frustrating. The Internet Archive offers a Vault product, where you pay $2-3/GB one time for perpetual storage into the future of your collections, items, media, etc. This pricing was previously predicated on the cost decline curve of storage. It's gotten a bit more expensive due to the AI bubble inflating the cost of storage, but to your point, the Internet Archive is a superior custodian than some rando tier 2 cloud provider who won a contract from a non profit and will lose all of the data when they go bust.

    https://archive-it.org/vault/

    (huge fan of Stokes as a fellow archivist, recommend the documentary on her: https://en.wikipedia.org/wiki/Recorder:_The_Marion_Stokes_Pr...)

ktm5j 3 hours ago

> This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.

Okay hang on now.. that strategy comes with its own risks. Risks that I imagine PBS was trying to avoid by using a third party that specializes in archival data storage. This is a silly takeaway.

Edit: I was too quick to comment. I think they are suggesting that the NAS should have been used in addition to the third party storage, which is actually a great takeaway.

  • jerf 3 hours ago

    I read it as a statement of just how little data this is by modern standards more than anything else. This company is going out of business and PBS is losing this valuable data, and there's legal drama and multiple corporate entities involved, and if you read all that you might think this is some amazing amount of data that is actually difficult to deal with. Instead it's something I could stroll down to Best Buy right now, plonk down my personal credit card, and walk out with enough stuff to do a credible backup job myself, and we're not even challenging my credit limit or anything.

    10 years ago, this was a lot of data. It isn't anymore and it makes it that much more bizarre that they're having that much trouble getting it back. It would be less to just give them access and let them download it themselves and perhaps stick it somewhere else temporarily than to pay lawyers to deal with legal filings.

    To put it in vaguely physical terms, what once was a university library is now a good solid bookshelf full of stuff. Legally wrangling over a huge library is one thing, but if it's just a bookshelf, metaphorically, let them come in for an hour and clean it out themselves rather than bringing in the law. Obviously not a perfect metaphor on data size, this is more about how lawsuits are more of a hassle then just giving it to them.

    • ameliaquining 2 hours ago

      The reason nobody's just giving it to them isn't because the amount of data's too much of a hassle, it's because nobody can figure out who has the keys to the now-defunct storage company, and Iron Mountain quite reasonably won't hand out their customer's data to a third party without a court order.

      • jerf 2 hours ago

        Yes, I agree completely. It just heightens the pathos of the story that it is over what is now such a small amount of data.

    • mindslight 2 hours ago

      > 10 years ago, this was a lot of data.

      More like 20 years ago (sorry). 10 years ago this would have been slightly stretching, but not by much - 11x8TB raidz3. Shuckable drives would have cost $3300 (plus the chassis), and already out of the husk probably 1.5x more. So let's say $5500 total even back then. (I built my current array around the same time)

      Funny, these days you could do it with 4 spindles (4x24TB raidz2) but with the rampant price inflation in storage, you're still looking at $4500.

      • js2 2 hours ago

        I recently pulled up my resume from 25 years ago and came across this amusing (by today's standards) tidbit: "Re-located a web-server farm between two data centers with zero downtime. This included migrating over 150GB of data between three Network Appliance Filers and moving more than 12 racks of equipment."

        A whopping 150GB!

  • MostlyStable 3 hours ago

    Having that be there only backup solution would have been silly and dangerous. Having it be part of their overall backup solution including with something like the provider would have been smarter and better and prevented this entire problem

  • ndiddy 3 hours ago

    > It's not clear if the channel has backups of its data, but based on its determination, we suspect not. This is strange, though, especially for a major organization, as one can easily purchase a six-bay NAS on [Amazon] and populate it with [the best HDDs].

    I think they just put that stuff in so they could shill their affiliate marketing articles rather than out of any actual concern.

  • thebitguru 2 hours ago

    The point still stands, but in what world can you build a 50TB NAS for $1,000 these days? I want to go there

    • dghughes 2 hours ago

      Yeah I bought a five bay Synology NAS two or three years ago I think it and three 18TB WD Red drives total cost $3,000CAD? I was going to save up for two more drives. Now each drive is $1,000 CAD a 150% jump. edit: I checked $1,350 each so about 250% jump ugh!

    • nairboon 2 hours ago

      $1000 is for the NAS, they still need to populate it with a couple of drives.

  • mcmcmc 3 hours ago

    3 copies, 2 different media, 1 offsite. One backup is none

    • anonymars 2 hours ago

      Importantly, "one cloud vendor" is also none

    • embedding-shape 2 hours ago

      1 copy, one other media, is better than 0 copies. Clearly 1 != 0.

  • echoangle 2 hours ago

    Technically you're still correct even when you would advocate for both. Every additional copy increases the risk that the data is accessed by someone else.

    • mulmen 2 hours ago

      What risk? The point of PBS is “other people access the data.”

  • OutOfHere 3 hours ago

    A NAS is at risk from malware that can erase all of it. 100GB Bluray discs exist and work more easily for archival storage, as does multi-terabyte tape drive storage, both of which can stay disconnected At least two offline copies, combined with error correction codes, are advised. A cloud backup is also advised.

roughly 2 hours ago

Calling out Nine PBS for not accounting for the fact that their vendor could effectively just decide to commit crimes to get out of their contract and their vendor’s vendor could just decide to compound the problem in the face of a court order is wild. Yes, we’ve all learned to anticipate everyone else on the planet being an absolute shitheel; no, Nine PBS did not do anything wrong by assuming the professionals they were paying to render a professional service to them would do so. We’re all paranoid engineers, but this is absolutely no goddamn way to run a society.

dehrmann 2 hours ago

> since 2019

At that point, why wouldn't you pick a major cloud vendor? This feels like more of a governance failure.

adolph 3 hours ago

Not PBS but an affiliate broadcaster and the data is not lost as in destroyed, but legally encumbered.

Full headline: PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct — public TV channel sues Iron Mountain data center, which hosts archival materials, to ensure preservation

The channel’s attorney also reached out to Iron Mountain to inform the company of the pending case in Colorado; Iron Mountain finally admitted to holding the data. Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.

  • ApolloFortyNine 32 minutes ago

    > Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.

    This does make sense. I'm guessing the first request some rep just went sure its you're data why not, but in reality, it's OSS's data, and you paid OSS to store it. It's also likely encrypted in someway.

    TLDR still, is especially when it's a reasonable amount of data, just have more than 1 backup.

    • RevEng 8 minutes ago

      I store my belongings with a storage company. They rent facilities from a property owner. The storage company goes under while my stuff is still in the storage unit. I go to the property owner and say my stuff was in there. They say, no, that's owned by the storage company. Incorrect - the belongings were owned by me the entire time, the storage company was just holding it for me. I need to prove it was mine, but I don't lose property rights just because it's in someone else's possession.

      Likewise if I store data with a cloud provider. Sure, Iron Mountain shouldn't just offer it up without proof, but if I give them notice of a legal case they should be sure to hold on to it, and if I win the case they need to hand it over to me. All I need to do is provide proof that I am the rightful owner of those goods.

Good4boothee 2 hours ago

Isn't 50TB just 6 years of videos, in even the best case (1GB/h)?

  • undersuit 2 hours ago

    This local PBS wasn't playing it's own footage 24/7. The data in question could be just their local segments, an hour or two of local coverage and maybe their fundraisers.

ck2 32 minutes ago

even r/DataHoarder won't be able to help that one

RIMR 2 hours ago

From the article:

>This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.

Look, I know that a certain percentage of Tom's Hardware articles are just sales pitches for a NAS, but this is asinine. Iron Mountain deals largely in tape storage. 50TB of tapes fits in one of their small archive boxes. If they're using the latest tech (LTO-9), it fits on 3 tapes.

This is exactly how you are supposed to store archival media for decades, not on a NAS. A NAS is for storing the files you need to actively access.

blltprfmnk 3 hours ago

Title should be updated to reflect the fact that this is a single PBS station.

  • cootsnuck 3 hours ago

    That made me feel relief until I realized it's my own local PBS station.

    • toomuchtodo 3 hours ago

      Any interest in connecting me with someone there so I can attempt to obtain a copy for the Internet Archive?

  • ButlerianJihad 3 hours ago

    Ah, "Channel Nine PBS", not a cluster of nine pbses