/hys/

Hoarding general

Quick Guide:
clickcomfy.org/data/

Recommended Drives:
Samsung EVO SSD
Western Digital Reds
Western Digital Enterprise Yellows

Other urls found in this thread:

nctritech.wordpress.com/2017/03/07/zfs-wont-save-you-fancy-filesystem-fanatics-need-to-get-a-clue-about-bit-rot-and-raid-5/
unix.stackexchange.com/questions/18154/what-is-the-purpose-of-the-lostfound-folder-in-linux-and-unix
hydrusnetwork.github.io/hydrus/
hydrusnetwork.github.io/hydrus/help/index.html
hydrusnetwork.github.io/hydrus/help/faq.html
hydrusnetwork.github.io/hydrus/help/access_keys.html
mediafire.com/folder/yoy1dx6or0tnr/tag_archives
ipfs.io/ipfs/QmWV8dpvs5LGpmCo6n7FUcZgXUFVD2jLTeSFGCYMaxN7gi
newegg.com/Product/Product.aspx?Item=9SIA5AD5VX9593
newegg.com/Product/Product.aspx?Item=N82E16822146134
newegg.com/Product/Product.aspx?Item=N82E16817182316
newegg.com/Product/Product.aspx?Item=N82E16817182247
twitter.com/NSFWRedditVideo

Guys don't use link in OP, literally just a list of amazon affiliate links.

I pirated bo1 days before it came out and got a strike, got another one for downloading bo1 repack last fucking week who cares reeeeeeeeee

What vpn

Post good sites for downloading anime

>tfw live in third world country and don't have to deal with strikes

for the first time in my life it feels good to be a lesser human being

If you're talking about an HDD, ZFS does that for you.

Strikes? What's that? Your ISP punishes you for pirating? Damn. What happens when you get three? Do they cut off your internet while still taking your money? How evil.

>he doesn't have access to AnimeBytes

Hard drives also do that and have done so for a long time. This is a nice post on the subject: nctritech.wordpress.com/2017/03/07/zfs-wont-save-you-fancy-filesystem-fanatics-need-to-get-a-clue-about-bit-rot-and-raid-5/

Comcast likes to send me notices about my INFRINGEMENTS. I'm at 3 strikes and I was throttled from about 3MB/s to 800kbps

>private trackers

>Comcast
There's your problem. Move to somewhere with a local or regional ISP cause they don't give a shit.

>muh sekret klub

What's the best price/tb in a reliable hdd? Looking at the used HGST megascale series, since they are $25/tb.

>It’s like the “knowledge” that you’re supposed to discharge a battery as completely as possible before charging it again which hasn’t gone away even though that was accurate for old Ni-Cd battery chemistry and will destroy your laptop or cell phone lithium-ion cells far faster than if you’d have just left it on the charger all the time

Oh no, I've been following old knowledge and never updated myself

My torrent drive just got fucked for the 5th time now but chkdsk /r usually fixes the problem -- how much longer can I keep doing this before I'm fucked for good?

Can I just get a USB to SATA converter and turn any HDD into an external hard drive? External hard drives seem overpriced.

Just download anime. American ISPs don't give a shit about anime, they only care about the media they own and their friends own.

You might want to consider a NAS running Linux or BSD and using a better filesystem like ext4, XFS or btrfs. NTFS is an ancient and terrible filesystem.

>Hydrus Network

So, what the fuck is this?

Some kind of personal booru or you can get stuff from other people?

We really need to share some hoarded shit

The kikes realize it's not worth it when the vast majority of 3rd world normies still buy pirated CDs and DVDs.

TIP 1: Use dupeGuru, FSlint or Fdupes to look for duplicate files.
TIP 2: If you want a NAS go for OpenMediaVault instead of FreeNAS for better support.

Thank you; I was actually thinking of setting up a server running Debian anyway so I'll definitely be taking your advice.

If this general lasts, somneone do a non retarded one for next thread, this OP is a subhuman trash and his link is expencing us to hoard stuff on SD cards with referal links and buy WD crap

I buy refurb Hitachi 3TBs from Newegg for like $20/TB

It keeps image databases with tags so you can sort them.

How do they hold up? I've only used new hdd's in the past, but commercial grade new equipment is expensive.

I have four of them in a ZFS array. it's been over a year and they've not coughed up any checksum errors, so I trust them

Sounds good, but the cheapest one I've found is $70 with $10 shipping.

>ext4
I switched over to using ext4 on when I switched over to linux.
Ive been finding files in the lost and found folders, shit that I actually want and know for a fact I didnt delete.
is this a common occurrence on ext4?

You should probably check the health of your driving with some smart software. I use smartmontools from the command line but things like gnome disks have guis for it. It sounds like your drive might be dieing, I've honestly never had anything actually in my lost+found folder.

Jesus man, definitely sounds like a drive failure.

huh, yea probably, good point. About the time I found that was about the time I migrated to my new drives and I haven't seen anything in there since then.
>check the health
Ive monitored smart data with gnome disks but it always comes back "healthy" even when i know damn well the drive is dying. iirc Ive used smartmontools before and got a clean bill of health aswell.
Ive already moved all the data off of one of them, and used it to back up the other, so the data is at less of a risk than just having the one copy.
When a set of new drives come in, I might stress the defective drives to see just how much more they can take.

Run fsck & it may recover them. They might be from a crash or power outage.

source:
unix.stackexchange.com/questions/18154/what-is-the-purpose-of-the-lostfound-folder-in-linux-and-unix

>They might be from a crash or power outage.
That might be it aswell.

would you happen to know about recovering a lost or damaged filesystem?
the tldr is Im a dumbass and now 3/4 partitions show up with their files and everything in testdisk, but partition 1 wont show up and I fear intel raid overstepped and erased.

>would you happen to know about recovering a lost or damaged filesystem?
Research fsck. For anything more extreme I would immediately shutdown the system then reboot into clonezilla, a cd-booted duplicating OS. Alternately to clonezilla dd if you're comfortable with it. Use dd best from an OS that is run-from-ram such as systemrescuecd or puppy where you can dd it without mounting (this to preserve further damage).
Finally if the system can't be recovered, photorec is another (loses filenames) alternative to testdisk.
photorec and testdisk have worked wonders for me in the past.

yea, I unplugged the drives (raid 1, 2x4tb) so that I wouldnt be tempted to fuck with them. Right now they are hooked up but unmounted so that I can pull random files at will with testdisk.
The issue with photo rec is the loss of file paths, since the partition contained about 700gb of virtual machines and vmware likes to have harddrive images split into multiple files. I will never be able to put the right pieces back together if I loose the folder they belong in.

I read something about a backup up of the file system with some types, but cant find much info about it. I also think it might have something to do with what sector its seeing the first partition start at, its saying it starts at sector 0, but that I think the beginning of the drive would have had some info about the raid there and the real partition 1 starts somewhere else

Im going to need more drives before I can begin unsorting this mess. Does fsck make changes by default? And yea testdisk and photorec are life savers
Thanks for the info

Okay, I still have an unused external hard drive that needs to be used. Due to cross platform demands, I'm unfortunately limited to NTFS or exFAT. Which one is more likely to fuck with my files? I don't want to rar everything with error correction.

>cross platform demands
You are stuck with NTFS, tough luck kiddo.

Gotta hoard more...

Read
hydrusnetwork.github.io/hydrus/
hydrusnetwork.github.io/hydrus/help/index.html

What informational data do you like to hoard?

I've got a small collection of articles, papers, tutorials, and interesting forum & messageboaord posts.

Just use smart(montools) to run a long/extended test. The result of that is quite reliable in that it should find drive defects.

implying that is hoarding
Thats just a bit of clutter

will do, thanks.

Scientific papers, magazines, textbooks (lots), video tutorials (from Lynda, Pluralsight, VTC). "The All-Embracing Library" sums up pretty well.

Also bookmarks (over 30MB) and RSS feeds. Both underrated.

Hydrus shares "hoarded" tags primarily.

You can share files with it but thats not a focus at this point. It can download files from various image websites though.

Thats fucking cool

It is an amazing end user data hoarding tool.

The "downside" for people who aren't yet hoarding data is that it "messes up" their prior organization. Hydrus doesn't use existing filesystem paths but rearranges everything into a hash-based folder / filename structure like most "cloud" storage stacks also do.

Do people really watch anime or is this some kind of meme?

>Hydrus shares "hoarded" tags primarily.

How?

pantsu.cat and nyaa.si

This whole website exists because of anime. Any more questions?

See or the previous thread where it was explained.

It works by matching databases containing hashes and tags against the hashes of the files you have.

The sharing part is primarily done with "public tag repositories" - PTR. AFAIK there's just the one big official one, but anyone could start 'em.
I've committed many chinese porn cartoon tags to the PTR myself, if I tag my files anyway I might as well share the objective ones, right?

But you can also create (scrape from *booru, enter into hydrus) and share database dump files which users can then add to Hydrus - it'll use them to tag files as well.

hmm, that is a downside considering I have a massive collection already sorted into folder categories. Thanks for the warning.
Youd thing it would be able to keep track of the files checksum and you can assigne it a parent directory to monitor so that way your organization isnt fucked and it just tracks everything in a database instead of moving stuff everywhere

PS: The PTR itself is a database that just can be sync'd regularly.

It's equivalent to Hydrus doing a "git pull" or something on that database file, it'll download the updates and apply them to the local PTR database that your client then uses. You have a full copy locally.

building a nas is as expensive as the desktop that i have
is there a reason at all to build one from scratch or should i just get more hdd for my desktop and store everything there?
i just want to be able to store all the movies that i torrent from private tracker and watch it without much seeking delay

> Youd thing it would be able to keep track of the files checksum and you can assigne it a parent directory to monitor so that way your organization isnt fucked and it just tracks everything in a database instead of moving stuff everywhere
That would be a possible design choice but you'd have a lot more DB lookups that way because you'd have to do an additional match-hash-to-filename step.

Plus you might want to keep exact folder indices just so you can detect moved files without it taking ages or the user having to explicitly move them from within the client or whatever. And more.

It's an arsepain. Hydrus' approach of letting the filenames be meaningless is better if you ask me. Basically, I recommend to just do this migration and abandon the folders (you can regex them into tags) and file names.

Ah yea, the dev also got a FAQ why he forced this:
hydrusnetwork.github.io/hydrus/help/faq.html

How big is the main PTR now?

Well, if your desktop happens to be running Windows one of the reasons to do another NAS box would be to make use of Linux (mdadm) RAID, Ceph or ZFS which all can make operating a bunch of drives easier and safer against hardware failures.

Another reason might be that you do a less power consuming storage box for torrenting, running a 200W-when-idle monster might cost more over time than some 20W-under-load torrent box. Ditto if it's the file server for multiple devices, like your HTPC and main computer and smartphone. Your main computer may not always have to be turned on.

But the simplest way for now might be to add the drive(s) to your desktop, sure.

>Hydrus' approach of letting the filenames be meaningless is better if you ask me
for stuff like my 10gb folder of "shit to sort later", I would have to agree with you. But when I assign file names it usually involves tags and a descripter like "that one bitch wearing x and y in the place doing this weird shit.jpg" and Ill use the search function to search the names for the tags.
naturally its not as clean as hydrus approach, but I also get to maintain the folder genre organization along with the separation of shit that Id have to kill myself if someone else saw

>Does fsck make changes by default?
fsck had an option (used in bsd), "-y" that automatically fixed errors. I'm sure in linux it's either the same or there's an equivalent, but haven't had to use it in linux yet. Also importantly, I've no experience with fsck + raid. So there might be some extra precautions to take.

good luck!

You can import the old filenames as local information (not exporting it to PTR) and try to use it that way.

> folder genre organization
Waste of time deciding multiple folder conflicts or trying to remember what other genre you might have assigned THAT file you're looking for to.

> separation of shit that Id have to kill myself if someone else saw
Sounds like you could just keep that in an encrypted storage second instance or whatever.

Hm, I currently don't actually know. Several hundred MB for sure, but not sure exactly how much. I don't think I can be arsed to play with sqlite, so I'll leave this to you to find out if you care.

I do know I have about 11GB of older db archive dumps from *boorus and such - these were produced by people crawling the sites and sharing the resulting tag databases individually.

But the hydrus dev made the archive format for the dbs more efficient at some point, so that's not a terribly good indicator how big the PTR is, plus obviously tags get entered into the PTR independently.

true, I've been trying to find something that would consume less power and didnt know it can operate at 20w
maybe i'll take a look into it
my desktop is mainly a torrent machine and to play OW anyway so maybe i dont need a desktop afterall atm

I have a fuckhuge folder with tons of images saved from the boorus, is it worth downloading the big PTR and hope it tags most of them?

Thanks, Im going to need it.

Anyone save websites? There is usually issues using pages saved just through a webbrowser (save page as...). Used to use HTTrack but something about it bugs me.

>true, I've been trying to find something that would consume less power and didnt know it can operate at 20w
Various NAS can (though every operating 10TB drive will add like 7-10W - ish).

Another, even cheaper approach is to make one "torrent / file sharing box" with anything from a Raspberry Pi or Chinese HTPC or old smartphone or your access point (quite a few have usb ports and powerful enough firmware).

If it runs 24/7, it might very well cost less in power.than running a 200W monster gaming desktop or whatever.

Yes, definitely. The PTR is what you want most - It'll tag a whole lot of them.

It's probably even worth download the additional individual *booru dump files, but you can also just add them later.

>Waste of time deciding multiple folder conflicts
youre 100% right on both of those things. Ill look into this more.

>It's probably even worth download the additional individual *booru dump files

Link?

LURK MOAR or leave. Seriously where the fuck do you think you are? Look at the websites name, it's FAQ and all that other shit.

why would someone look at the type of website or read any of the information or rules before making a post on the website?

PTR:
hydrusnetwork.github.io/hydrus/help/access_keys.html

Tag archives:
mediafire.com/folder/yoy1dx6or0tnr/tag_archives (yup, most are older, they don't get re-crawled by users monthly or anything like that - it's just whatever someone felt like crawling a while ago)

It used to be so they don't get assraped. Guess people don't care about anything anymore.

Wget has many options to save websites, it all depends on becoming proficient at it. I know what you men with HTTrack, somehow is not the same.

the entire staff got fired and replaced by people from a different site that hates us, so yea its fair to say not too many people give a fuck anymore.

>the entire staff got fired and replaced by people from a different site that hates us
Thanks for the reminder.

(cont'd)
I got a bunch of tag dumps for mainly drawn / CG images from various threads on the Hydrus boards. Seems like a bit of a pain to find and link, so I guess I'll share by IPFS:
ipfs get QmWV8dpvs5LGpmCo6n7FUcZgXUFVD2jLTeSFGCYMaxN7gi -o client_archives

# to keep this from getting gc'd and share too:
ipfs pin QmWV8dpvs5LGpmCo6n7FUcZgXUFVD2jLTeSFGCYMaxN7gi

# web based download, unfortunately not gonna help with p2p:
ipfs.io/ipfs/QmWV8dpvs5LGpmCo6n7FUcZgXUFVD2jLTeSFGCYMaxN7gi


Obviously you might also have to run ipfs init and/or ipfs daemon.

fuck now I feel too.. time to suppress it with regret for wasting my life here.. yea anger is easier to deal with.

...

P-please reply again.

I'm lonely

kek you idiots.
Back to topic. Just a little question. I just heard good stuff about the HGSTs, but OP didn't wrote them in his OP. So are the Reds any better than the HGSTs? I was looking for the "HGST 0S04005" because it's only around 120€. The Reds cost a bit more, but it's still a good price I guess.

>kek you idiots.
Did you think those are 4 individual poster? 3 of them are me.

So, I think this drive is having some failures but looks "good" on health and it have no bad sectors (checked with HD tune)

Whats up with this motherfucker?

No I realised that.

wtf are you putting all of these hard drives in? external hard drive array?

w-what?

>dont watch anime except a couple shows that are long dead
>dont watch movies
>dont watch tv except a few shows
>dont like most video games
what the fuck do i hoard?

things you will never watch, listen to, play, or look at. hoard just to hoard.

webpages

>Western Digital Reds
>Western Digital Enterprise Yellows

A question for hoarders - how do you manage off-site backups (or do you?)? I have ~4 TB of data on my home server; uploading that would take forever with my 10 Mbps up speed.

Should I copy it to an external drive and then keep that drive somewhere else? If so, how should I handle keeping that backup up to date/doing iterative backups in the future?

Or...?

Get a external drive or a dock station and connect your drive to the other computer.

These WERE the best deal for new HDDs for about the last six months. Finally went Sold Out about two weeks ago. I got 6 of them in three separate purchases. Hoping they get more in stock; would like to have another 4 or so.
- newegg.com/Product/Product.aspx?Item=9SIA5AD5VX9593

By BackBlaze numbers, these specific HDDs have a fail rate of between 0.4% and 0.6% ... nothing beats them, and I've not seen anything that quite matches.

As far as refurbs, Newegg had 2 TB versions of these for $35 a few weeks back. Can't find a link to the Megascale 2 TB right now, they might be sold out, too. Closest is the Ultrastar 2 TBs, and they start at $40 and go up to $45:
- newegg.com/Product/Product.aspx?Item=N82E16822146134

I prefer the Megascale drives; they're 5900 rpm instead of 7200, but you really can't tell the difference in practical use. My Megascale HDDs (the new 4 TB ones) run about 28-30 degrees C, while my Ultrastars (2 TB refurbs) run closer to 35 in the identical environment. Those are perfectly acceptable temperatures, but the higher platter spin + temp + being refurbs makes me more cozy with the new 4 TB Megascale for important data. I've been using the Ultrastar refurbs for duplicate data and in-use to stream music & video on my daily driver.

Things related to stuff you like?

I got two of these for $30 each recently:
- newegg.com/Product/Product.aspx?Item=N82E16817182316

You can put any 3.5" SATA HDD in them, they have a cooling fan, and they have both USB 3 and eSATA III connections. I usually use the eSATA cable for speed & reliability, but the USB 3 is pretty much just as fast and is a good quick-connect to hop around different computers that don't have eSATA ports. Comes with both eSATA cable & USB 3, plus a bracket to install in your computer that converts one of your mobo SATA ports into eSATA.

The reason I picked two of these up when they went on sale is that I bought one when they were first introduced as a new product ($20) about 4 years ago, and it has been reliable, cold & silent forever. I've probably swapped 20 different HDDs in it over the years, and (with some creative ghetto engineering) also put 2.5" HDDs and SSDs in it for temporary transfers & recoveries & such.

You can get cheaper enclosures, under $20 Rosewill are common and reliable. But, you don't get that cooling fan and they usually don't have eSATA.

These after market external enclosures are about the same difficulty level as installing an HDD in your computer. So, very simple. Of course ... you don't want to shock them if at all possible. When handling them, pretend they are a bare drive because that's very nearly what they are.

These are also excellent:
- newegg.com/Product/Product.aspx?Item=N82E16817182247

USB 3 + eSATA + fan, also comes with mobo SATA to eSATA bracket. They used to be cheaper, like about $28, but the price has crept up over the years. Watch for them on sale. I've got three of those around the place, too, and years of great service out of them.

This
Probably the reason why the op is so sparse, because this general's made by a shitskin.