Lets have another thread about Sup Forums shit hoarding 7 million loli images and home servers

Lets have another thread about Sup Forums shit hoarding 7 million loli images and home servers
Maybe this time someone share something

And while we are at this
How can I self host or plain host and run this shit to look for old stuff?

archive.org/details/bui_pm_archive

Other urls found in this thread:

my.mixtape.moe/irnjwj.rar
myfigurecollection.net/
archive.org/details/bui_pm_archive
web.archive.org/web/20070709230544/http://jj.am:80/gallery/v/Amateur/
twitter.com/AnonBabble

There is literally nothing wrong with having 10TB of loli porn

>Maybe this time someone share something
someone wanted to share but nobody could agree on a good way to share it

SFTP = uploader gets their home connection trashed for weeks/months
Torrent = Files can't be updated, static
DDL = no file host will allow 1TB+ without paying some major cash

I have an idea though.
What about a paid VPS with around 500GB of storage? Someone can upload 500GB worth of content to it, then simply allow other anons to access it. The user gets to keep their upload bandwidth while everyone else can download at maximum speeds.

He wanted to share 20TB, so no

The sync P2P things was a nice way to go

This
> have local file server with many websites and file directories loaded with all sorts of stuff accessible through apache
>lol just open your ports user!
> open port for 2 seconds, server ddossed instantly
> close port, network connection ddossed for a fucking month
Never again, fuck you fags.

>20TB
yeah, there's no easy way to just share that large amount of data.

Assuming they shared 1TB(ISP cap) a month, it'd take almost 2 years of constant uploading for 1 person to finish.

yeah that's why I wouldn't recommend using your home connection to share such a large volume.

I had a simple apache server running on a vps which easily handled 5k users downloading files in the past. Since it's 20TB, it'd have to be split into smaller chunks. There's no reason all 20TB needs to be shared at one time.

The only problem with the VPS method is anons who get left behind. When the majority have downloaded the first 500gb chunk, then they'll forever miss out on downloading it again since the next chunk needs to be uploaded.

>The TS-1635 is equipped with an Annapurna Labs, an Amazon company Alpine AL-514 quad-core 1.7 GHz ARM® Cortex-A15 processor

>(ISP cap)

Even if they had unlimited, most US connections don't offer fast upload. I doubt they'd even reach 1TB in a month.
I think GoogleFiber is the only ISP which offers true fiber speeds, which like 0.1% of the population owns.

I think this would have to be operated on a more personal basis. Post things you have and have people request just the things they want, and have access opened for them, instead of just having everything open for everyone.

nah bruh. I'm on Verizon and have 100MB/s both up and down. They also offer gigabit up and down as well. Verizon, Optimum Online, and a couple other carriers around all started offering equal up/down plans and all the prices dropped ever since Google got into the game.

> Post things you have and have people request just the things they want
and if they want everything? because most of the anons in the last thread didnt request anything specifically, just wanted 10tb+ dumps

I did actually dump my oldest chan folder by the end, but it 404d shortly after.

>mixtape.moe/irnjwj.rar

Both of those ISP's are NorthEast coast only though, particularly NY.

For the large majority, only Comcast and ATT are available.

Okay then lets request Sup Forums stuff that just can't be found anymore
Someone with some archive of old chanwhores to share?

my.mixtape.moe/irnjwj.rar

Fixed url for ya

i only have hentai sorry

how did it die that quick? how big was it?

I really enjoyed the days of p2p like lime and frostwire before torrents effectively replaced them. Maybe its worthwhile to start another oldschool p2p network.

I am literally downloading the entirety of myfigurecollection.net/ RIGHT NOW

I'm thirsty for more sites to rip.

Appreciate it user, I've been here forever and still dont know what links will flag my post as spam, so I never type them in full.

Not very, its only around 80mb, but most images back then were smaller or compressed too.

>And while we are at this
How can I self host or plain host and run this shit to look for old stuff?
>archive.org/details/bui_pm_archive


No one knows how fuck can you do anything with the archived archives dumps?

I want to host it and scrape it or something

...

i dont even know what the fuck that is

...

ä

Are you okay?

>I'm thirsty for more sites to rip.
If you're into data about old arcade games, arcade-museum.com is pretty good
There's a bunch of pdf manuals/information on a ton of cabs that can be rather hard to find elsewhere.

im not sure if its good or bad i've seen 90% of those images

Only bad part of uploading a bunch of cool stuff is that meme aggregation sites or anything that profits off old chan content will suck it right up, but whatever, preservation before consequences, faggots will lose interest eventually.
We should collect old screencaps, the oldest dated images we can find, even just the kind of dialogue happening back then. Im also up for even non chan related stuff you fags might have, anything great from the early internet.
As a guideline for newfags that would ideally be pre 2007-2008 hopefully.

Sorry, Hentai/H-games,H-content only.

I've already got 10TB+ dedicated to games.
1st gen~7th gen, all games.(NES~PS3)
Entire siterip of Doujinstyle
Massive chunk of PC games
Hyperspin(arcade program that has manuals/videos for every classic arcade game)

It's pretty fucking sweet.

Some old and closed archive with 167gb of Sup Forums images but I have no idea how to turn the 9 million images and the sql files into the website it was

are you sure 9mil files fit into 170gb?

170GB is a lot
I can give you a exact number when my computer managers to unrar everything

2mil images @ 500kb = 1tb

but anyway, what would you do with an old Sup Forums archive?

>@ 500kb
Images are not 500kb
56kb internet ages images was not fucking 500kb
I mostly want to host the archive itself to be able to search fo some topics and archive them

>56kb internet
Sup Forums was built in 2003 but didnt take out until 2005.

even in 2003, most people definitely had dsl internet.

> host the archive itself to be able to search fo some topics
what topics?
what could you possibly find of interest from old Sup Forums aside from memes and shitposting?

>.tar inside a .tar.gz

Is this hell?

anyways this is whatever fuck is inside db, what am I supposed to do with this?

My main reason is simple. is a 2009 to 2013 archive and we have 0 alive archives from that ages

also, looks like the actual images are "just" 2m
I will need like 4 hours to get them out of the tar

>what am I supposed to do with this?
Come out with your hands up

The internet is not going to close, stop wasting electricity and money on drives

TAR is used to archive (multiple files in one)
GZ is used to compress one file.

First your archive everything with tar, then you GZip the archive, hence your .tar.gz

Loli lewds or cutes? Important question.

First folder already have a ton of images google can't find anymore
But I just can't manually look 2m images

Both I guess since that guy said he ripped the entire "loli" category of nhentai

nobody wants loli cutes

wait what the fuck
why?
>mfw your post

Something about it being removed in some alternate reality or something

>Relying on "some other guy" to host your content indefinitely
Pleb.

Well then go and host your own imageboard

How about I just store local copies of shit I like?

You don't like Sup Forums?
Are you hiro?

Why fuck are servers supposed to have 32GB ram?
I'm running one with 2 just fine

who says they're supposed to?
most people buy a lot of ram for their server because they use zfs, which requires a ton of ram. there's also the fact they probably want it doing a bunch of shit.

>mfw its another thread where everyone talks about sharing but nobody actually shares

this is literally what AWS s3 and backblaze b2 was designed for.

You can start
Here, let me share some internets with you

Google and amazon unlimited clouds are dead for a long time since they ban accounts/encrypted upload/mount methos as they feel like it, I lost 71 rcloned TBs on google

I'm still lurking this folders
Damn, memmer jj.am?

soulseek is still working and useful

>zfs, which requires a ton of ram
no it doesn't. unless you use dedup, which you don't want to do anyway.

I still use soulseek and emule

>web.archive.org/web/20070709230544/http://jj.am:80/gallery/v/Amateur/

Where fuck was site downloaders when that site was a thing, so much lost

that's not what it is or what it's for. you're clearly just a moron.

Explain yourself in 15 words or less before I filter you

>emule
Mah nigga. There I still find things that aren't available in other websites.

even vps/dedi providers do usually have a data cap but they let you buy more as much as you can pay for.

kimsufi have no caps, I'm using it as a seedbox for years

>jj.am
Fuckin hell

>vps
If I fucking knew how I could do it I guess?
I'm the guy with 1GB internet and 90TB free and I literally have only running a idle torrent client here

how many lolibytes is that?

is anything other than what is on archive.org left from the first Sup Forums domain?

I got the files too just because but yeah I don't fucking know what to do with them

it says there that the speeds drop to 10m after 1tb if you dont pay for more.

The only almost cheap drives you can get over yurop are the 8TB Seagate ST8000AS0002, how bad are they exactly?
I don't really want a lot of storage, like a couple of this things would be fine for me, yet I don't know if they are good for raid5 or if I need 3 of the same drive/size for it?

Seagate Archive are shingled drives and they aren't good for RAID use AFAIK. Read up on it.

>4TB drives are more expensive here than that 8TB ones

Being yuro is suffering

wait, you have all 1st to 7th gen games on 10TB?
is just 10TB enough?

I have a 7TB collection of PS2 games so I'm pretty sure is not

>With ZFS, it's 1 GB per TB of actual disk (since you lose some to parity).
> For example, if you have 16 TB in physical disks, you need 16 GB of RAM.

>Depending on usage requirements, you need 8 GB minimum for ZFS

Avoid cheap Seagate crap and large drives.6TB+

If you want to do storage properly and not have it shit the bed in 2 years, buy enterprise drives. Enterprise drives cost much more, but they're built to last longer in an actual data center setting.

What about bittorrent sync?

I have 12 3TB disks in six mirror vdevs, which equates to 15.9 usable TiB. I set a config option in the kernel module to limit its RAM usage to 4GB. It works just fine. Massive amounts of disk cache are needed for good performance, if you're serving a bunch of users at once doing lots of random I/O. For one-to-three people watching anime, it makes no difference, the disks can keep up fine.

What are the chances of HGST drives getting cheaper on 2018

6th gen and up is USA only. Didn't bother with foreign duplicates, though I'm sure there's around 200~300 JP only games that I'm missing.

1,850 games total for USA = 3.5TB(Other half on another hdd)

I was thinking about buying a bunch of those 3 TB refurbished drives for the time being and setting up an encrypted raid 6 array and pick up new drives as time goes on.

I haven't setup anything this complicated before and had some questions.
1. Which file system should I use? I was looking at ZFS but with drive pooling I don't think you can resize, add, remove things. Because of this I was looking at XFS using mdadm for RAID. Is this a good plan or should I do something differently.
2. I want to have all the drives encrypted. Is there any overhead I'm overlooking? I just have to set them to auto decrypt from a keyfile in /etc/crypttab then set them up in mdadm using their respective mappers. I believe crypttab comes before fstab so I shouldn't have any problems there.
3. Data corruption. What will my filesystem and RAID array do in the case of data corruption? Will it detect it or have a way of preventing it?
4. Currently I'm working on a C++ program to hash every file in a directory recursively and save it to a text file, then compare it to the already existing dump to look for changes that way once I have a backup solution in place I can restore corrupted files. I'm kind of shit at programming though and my program took about 2 hours 30 minutes to hash 800GB of data using sha256 (I'm timing it using md5 right now). Is this any good speed wise? If I'm going to save a bunch of data it needs to be as fast as possible. If anyone wants me to link the source just ask.

Anything I should know or do? I just want to indiscriminately save everything. I'm also willing to do a lot of reading so I would appreciate any resources.

S3 and B2 are content delivery networks used for image and data backend hosting.

>archive.org/details/bui_pm_archive
Can someone seed this. I have 0.07% left.

Stop watching cartoons and grow up

You have a DDL option available you know

The torrent comes without the images files I think, just the db

You know what to do with it or you just want to browse 2 million images randomly too?

>48 bucks
>3TB

I'm so jelly of murrikans

for 4, I thought MD5 would be all you need for file integrity, and it's quicker

1. The easiest way to do incremental expansion with ZFS is to use mirrors, so you can add or upgrade disks two at a time. But they stick you with 50% space efficiency and you still can't remove vdevs once they're there. mdadm can have any filesystem you care to on top of it, but it doesn't give you the same checksumming and data-integrity guarantees of ZFS. It can't, because the way ZFS (and Btrfs) do that is by not caring about the layering-violation stuff, and having the filesystem and the RAID/block layer being in cahoots.
2.) There'll be some overhead but its minimal on any modern CPU (AES-NI and all) You can put ZFS, Btrfs, or mdadm on top of dm-crypt devices just fine.
3.) ZFS and Btrfs can find it (thanks to checksumming) and correct it automatically as long as there's sufficient redundancy. This latter part is the difference with mdadm, because they don't care about layering violations, the filesystem can say to the block layer "hey, bad checksum. Give me that same thing again, but from another disk/rebuilt from parity" mdadm has no way to pull the same trick since the filesystem can't talk to it - mdadm neither knows nor cares about the filesystem layer.
4.) Why are you doing that? People way smarter than you or I put this in ZFS and Btrfs for you. If you really want to do it yourself on top of one of those two, you can use parchive.

you can't tell me what to do

Yeah I don't know too much about hashing algorithms and I figured hurr durr sha256 > md5. But I guess the issue with md5 is someone recreating a file that collides with the hash. But that is not an issue for this case. I made a random data file using dd, hashed it using md5, changed a single byte to zero, then md5summed it again and the hash was different. So I guess it's good enough. I'm timing it on the 800 GB folder I have again but with md5 this time to see the performance benefit.

Main reason I don't want to go with ZFS is because it seems complicated enough for me to fuck it up and if I'm preserving a hoard of data I can't afford that. On top of that I don't have too much money to throw around so mdadm being able to just add and remove disks is pretty attractive. I want to be able to just buy brand new disks to replace the sketchy refurbished drives as time passes and possibly use them for backups.
Also the reason I'm doing the C++ project is because an user in a previous thread was asking about a a program that had this functionality so I decided to try it. I don't think it's very good, but it's mine and I like the idea of writing my own software/utilities. Also it's one of the more 'useful to real world applications' projects I've done since I'm still a student and can't do things that are too complicated atm.

>What will my filesystem and RAID array do in the case of data corruption?
Nothing, mdadm doesn't know how to recover. It can scan your array and detect corruption though, by default (on debian at least) it does so once a month.

I have a mdadm RAID6 array, 9 3TB drives. The way I do it is just backups. After the monthly scan I rsync things to my backup HDDs, which I keep offline and just connect to a USB3 dock when I'm actually doing the backup. Since my data doesn't change that much over a month, at first I do a rsync dry run and evaluate everything which will be changed. If something is fucky, I abort and have last month's backup safe. If not, I just go ahead and run the real rsync and update the backup.

The backup process is all manual basically, I have an interactive script which just goes through all steps and waits for confirmation before doing shit, something like:
>1. Identify drive under /dev
>2. Mount
>3. Check drive is right volume
>4. rsync dry-run
>5. Present all changes
>6. Present only files which will be deleted, once again (to avoid accidentally nuking shit)
>7. Actually run rsync to do the backup
>8. Show SMART report of backup drive
>9. Unmount/eject
>10. Start with step 1 for the next backup volume
It's not a very elegant way of doing it, especially since there's no incremental backup, but I have found it to be good enough for home use so far.

And what do you want to store there

It's nothing very unusual. Media, some backups of various PCs and the server's boot drive, some console games, a few GOG installers, source code and that's about it.

ZFS isn't hard user. Just read the documentation carefully. There's some excellent guides out there that explain how to get up and running. But you're right, you can't swap drives at will. Btrfs can do that, but for a long time it had a data-loss bug in the RAID56 code. That's supposed to be fixed now, but only as of kernel 4.12. Up to you if you decide its trustworthy enough, and you still have to run a fairly recent kernel for a while.

>i only have hentai sorry
That's ten times better than shitty 3dpd camsluts, I'm in.