Probably many of you are hoarding terabytes of data. How do you keep it organized...

Probably many of you are hoarding terabytes of data. How do you keep it organized, I'm particularly intrested in pictures.
For now I see 4 options available to keep them organized:

>filenames
+ multiplatform, does not require additional software
- limited by amount of max character per filename / path

>directory structure
+ easier than (1) to traverse within file explorer, multiplatform, does not require additional software
- needs you to create a fuckton of symlinks, because one pic might belong to multiple directories

>tags
+ pretty much the same as 1, but without limits and its' easier to search for particular files
+ multiple types of tags, ie. separate characters and artist tags
- significantly slower than 1
- requires a decent tagging software (and I don't know any decent one)
- requires you to store your pictures as TIFFs, which takes a ton of storage place and has no real advantage above ie. PNG for the sole purpose of storing pics containing undressed anime characters

>database or 3rd party solution (which probably depends on databases)
+fast
+all advantages of tags
-requires additional software
-which might not be multiplatform
-cannot search and files under file explorer

How do you keep your files or/g/anized ?

Other urls found in this thread:

hydrusnetwork.github.io/hydrus/
pastebin.com/di1DmvVG
w3.org/TR/PNG/#11textinfo
twitter.com/SFWRedditImages

Proper filenames most of the time.

Filenames. Then I can search for what I want. Hopefully I can transition to a filesystem which supports longer file names and trees within the next 5 years. (Looking at you ReFS)

...

Mind sharing your naming pattern?
something like:
wokada_(takouji)-vocaloid-hatsune_miku-luka_megurine-kagamine_rin-body_writing-no_bra-self_shot-vertical-underboob-twintails-high_res
looks like a mess, is hard to write and tends to be problematic with no-wildcard searches

>- needs you to create a fuckton of symlinks, because one pic might belong to multiple directories
I usually just pick one even if it can go in many

That's what I used to do in past, but apparently I couldn't find anything when I wanted to.

Filenames and directory structure. Exclusively, no tags or anything.

Pictures I took get put in folders named by date. (example: "2017-01-21 Description of thing I photographed") I've actually read an article or two on photography websites saying don't do this, their argument being that you'll want to find pictures but not remember when you took them. I've never run into that problem, myself, maybe I just think chronologically.

For images I save off Sup Forums I just put them into a folder by category. If there's an obvious way to name the file (say the image has text on it) then I do so.

I heartily recommend a search program called "Everything" to winfags. Someone wrote a Linux version called Angrysearch but I haven't tried it.

if that happens (usually to me it happens with porn) then I just pick one and leave an empty text file in the other directory saying "X video is saved in Y folder"

Thanks for your reply, user.
Atm the only type of files I cannot manage by putting them in some subfodlers are pics from anime and a few games (which I don't play, but Overwatch asses are nice too look at, arent they?)
I'm currently thinking about structure:
/{series name or game name}/{character name}/pics
as for the "main" sorting method, but there are a few characters where I have over a thounsand pics, also my "unknown" folder is growing at the alarming rate and I have no idea what to do about it.
I'll probably and up deleting a massive part of my collection (everything that isn't at least 8/10), but it's still going to left me with over a dozen thousands pics.
I thought about hydrus network, but I don't like the idea of uploading everything to get it tagged, especially with my shitty connection.

>I heartily recommend a search program called "Everything" to winfags. Someone wrote a Linux version called Angrysearch but I haven't tried it.
Yup, Everything is at least great-tier program, but since I want my solution to be as multiplatfom as possible (and independent of both filesystems and external programs), it's not gonna help in this case.

>Someone wrote a Linux version called Angrysearch but I haven't tried it.
there's also fsearch

well no matter how you decide to do it you're gonna have to go through all those thousands and thousands of pictures and add the organizational information. Be that a filename, a place in the directory structure, tags, whatever. You just have to sit down and start plowing through it.

the moral of the story is to not let yourself get in the habit of just dumping things into a big "to be sorted" area, but to organize them as you acquire them. same way it's easier to keep your room neat than it is to clean a messy room.

didnt read

That's actually not the case, I happen to made a several tries on sorting them.
I converted everything to tiff format, stripped it from metadata, renamed to md5 hash of file and started tagging with Photoshop Elements Organizer.
What I noticed is:
PEO is well made tagging software...
...but besides of that it's a garbage bloatware which created a miles long log file in my firewall, and I don't feel like keeping this trash on my PC just because I can tag pics with it. Also, it allowed me to tag files only on my main workstation with windows os.
Now I have about 80% of my collection in tiff, some tagged, some not and the rest is mixed filenames and fomats. I kinda regret doing what I did, but there's no comeback because I don't keep backup history that long (it started several years ago)
Now I'm looking for some better way to get things done, that's why I made this thread.
Spending a fuckton of time to actually sort/tag them is not a issue, I have plenty of spare time and I do love to go trough some of these pics from time to time

Does anyone here hoard data on blurays or other disc based media instead of a hard drives? I've been considering going that way since it seems to cost significantly less per GB with blurays until you get into storing enough data to use a RAID 6 with decent capacity hard drives. The initial cost for the equipment is also much cheaper (a bluray burner vs building a NAS) and there wouldn't be the reoccurring cost of replacing failing hard drives (though I don't know how often that happens or costs in the long run).

I'm planning on keeping some information on the data in the file name and keeping copies of the directory trees stored in text files so they can be examined and searched with grep without needing to insert the disc.

Depends. If you want to use it as a long term backup - sure, why not.
If you want to access the data stored quite often, I'd rather got myself a NAS. B Disks are easy to damage, pain in ass to use (you need to swap them if you want to get data from more than one at the time) and generally slower than NAS with even 5.4k drives inside.
Also, you can configure your NAS to keep your data redundant with raid 1 or some other software solution, while storing unique data on BD would require creating copies manually.

>Depends. If you want to use it as a long term backup - sure, why not.
Most of the data I store currently just sits around and is rarely accessed, which is why I started to consider blurays or other more archival focused options.

>Also, you can configure your NAS to keep your data redundant with raid 1 or some other software solution
I'd consider a RAID to be necessary since I wouldn't want terabytes worth of data I spent time collecting to become unrecoverable in the event of a single hard drive failing. From what I've read quality blurays are much more stable than HDDs, so long as you're storing them correctly, making redundant storage not as necessary.

>you're storing them correctly
That's one thing (and might be harder to achieve than you think)
>making redundant storage not as necessary
And that's
WRONG
Redundant storage is always a necessity, with the exception of storing data you won't mind to lose.

I have folders separated by show, and a bash script which uses exiftool to tag all the images in those folders with the name of the show. Now I want to come up with a way to do a reverse image search or something and scrape tags from boorus.

Mind sharing code? I had a similar idea.
Btw, what tool do you use to convert your images to tiff?

>That's one thing (and might be harder to achieve than you think)
Keep them in a box in my closet out of the sun. I already keep my room's humidity and temperature pretty stable year round.

>Redundant storage is always a necessity
If one HDD fails you lose terabytes of data that can take weeks to retrieve again. If one BD-R fails (which is significantly less like than an HDD failing if you're using quality BD-Rs) you only lose 25 GB of data that you can replace in a couple hours, and with BD-Rs data that can't be easily replaced can have a second copy made for significantly less cost than duplicating an entire hard drive. Redundant storage is not as necessary.

What about having a sqlite database?
>use "find . -exec md5sum {} \;" to generate list of hashes and directories
>put it to csv file
>import to sqlite table
>add tags table
>use md5 as foreign key

I think the biggest flaw of this idea is constantly having to update the database manually. And I don't really know how to deal with deleted files. It's not like I'm gonna keep list of removed files and then drop them from table. It's way too much work.

>can take weeks to retrieve again
naaah, it wouldn't
Anyways, I get your point. If it's really much cheaper than NAS then go for it, seems like a decent solution for your case

Directory tree is the easiest way.

that's horrible.

How is it horrible?

hydrusnetwork.github.io/hydrus/

This is pretty close to what you want.
You could also find the last version of Picasa and use that.

is right
It will work only if you have less than a few hundreds items in each directory.

I mentioned hydrus earlier, I might end up using it, but that's not the perfect solution either.

>It will work only if you have less than a few hundreds items in each directory.
It really isn't. It gets a bit time consuming to find specific images if your folder has a thousand images, but you can use tags if you really needed to.

Sure I'll put it on pastebin, you'll have to change the base directory to match yours. Also I don't convert to tiff jpg and png both support having exif info.
pastebin.com/di1DmvVG

thanks
btw, afaik PNG does not support exif tags

exiftool seems able to write them, it does complain about using nonstandard tagging but it still tags them.

intresting, I'll check it out

Just looked it up, png doesn't use exif but it has similar standard that lets you attach strings. w3.org/TR/PNG/#11textinfo

79 characters is max, still - good to know

>How do you keep it organized
I don't.

I do have a few tags folder in which I symlink images and doujins but I'm very lazy so it's not that useful.

I use the world's most advanced OS

>Devices: Windows
kek, enjoy your world's most advanced OS ™

a small 120gb SSD for games that don't work in wine

is your os advanced enough to do this? i didn't think so

snail is impure and a whore

Name 1 (one) os that can't.
Even wangblows lets you search files by tags, type, orientation, etc.
For example, if one writes:
orientation:gay
in explorer's search box, you are going to appear as a result

Memes aside, how is no options provided for user and no software making mac os "the most advanced" one?

your really dumb
don't reply to my posts anymore

>uses mac os
>shills mac os
>calls other dumb

man, I love this board

i told u not to reply to me anymore

OK

Movies are thrown into a folder with practically no regard to naming
Music sorted by Artist / Album / CD
TV Shows / Anime sorted by Series / Season
Pictures are being organized currently with no regard to file name but sorted by used IE reaction / expression.

>Movies are thrown into a folder with practically no regard to naming
same
>Music sorted by Artist / Album / CD
same
>TV Shows / Anime sorted by Series / Season
same
>Pictures are being organized currently with no regard to file name but sorted by used IE reaction / expression.
Aaaand here I am with my problem.

I guess I'll get rid of most pics saving only ones that are really god-tier, and then go with

../series/
character/...(pics)
...(pics)

plus some tags, auto ones for series/character (exiftool script) and some manually-added ones
I guess I'll go with either Picasa or Adobe Bridge for tagging, because first one is lighweight (sort of, at least) and second one is installed on my pc anyways, because my job requires me to use Photoshop from time to time.
All files will be converted to tiff, space shouldn't be an issue if I'm going to delete most of them.


That's my conclusion for now, I'm off to sleep. If anyone's got questions or some valuable input - I'll check it tomorrow and reply (unless thread goes to the archive)

your so dumb lol

I would say just dump them in a folder. It worked for me for at least 2 or 3 since I started "collecting" random images online. I only feel like organizing them because I have around 15 gigabytes of them.

Pictures folder: 132.497 items

Yeah, renaming is not an option. At all. I just move them to folders.