What do you use to find duplicate files, images, porn, etc. on your drives?

What do you use to find duplicate files, images, porn, etc. on your drives?

Other urls found in this thread:

hardcoded.net/dupeguru/
video-comparer.com/
mscs.dal.ca/~selinger/md5collision/
www58.zippyshare.com/v/j2xkenLO/file.html
virustotal.com/en/file/15f15086de102939941f2b0e784309f434810df6afe75ab2665ed0020bffd117/analysis/
nirsoft.net/utils/search_my_files.html
twitter.com/AnonBabble

Brain and eyes 'n shit like that.

fslint

look for matching checksums. then find similarly named files and inspect them manually.

hardcoded.net/dupeguru/

Dupeguru

/thread

>muh windows hate
lolz

developing for windows is a pain in the ass. it's not hate just because they don't accommodate babies.

>The last time I used Windows "for real" was more than 10 years ago, in 2005.
>I hate Windows with passion now. It seems to get everything backwards.

this is Sup Forums levels of 'i've never ever had it, used it, seen it or experienced it, but I HATE IT' lmao

its difficult to manage 2.5TB of porn and filter out duplicates when you have 10k+ images and 1k+ videos

fslint

I use my brain.

Awsome Photo Finder for pictures

Dup Detector

Developing for Windows is *the fucking worst*. Everything is so ass-backwards, fucked up, and overcomplicated.

Microsoft makes very few considerations for developers that aren't paying big money to suck their dick. This is Microsoft's entire business model. It is irrefutable. It was bad in 2005 and it's worse now. Go ahead and try to get started writing a quick program in C++ on Linux. Do the same on Windows. If you genuinely think it's easier to start on Windows, please go back to Sup Forums; you're probably a fucking idiot.

CCleaner has a Duplicate Finder under "Tools"

use btrfs and don't care about shit

ignore retards. this board is full of people whose use of technology pretty much ends at "collecting 20 reaction images"

This.
>download 1TB of visual studio

Explain I know it's fs

md5sum 2bh

Everything else is botnet

BRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR UFFFFF

*sniff*

VisiPics. A hash checker is one thing but this program is actually pretty good at finding images that look similar as well.

Bump

>object-oriented API
>best IDEs (Visual Studio, C++ Builder, etc.)
>most comprehensive documentation
>great support
How utterly incompetent do you have to be to fail at Windows development?

>he doesn't have a filesystem that supports deduplication

'Awesome Duplicate Photo Finder' is pretty good for finding matching images with different resolutions, haven't found any other tools that work as well as it

Is it open source?

Sadly I don't think so, and last time I checked I couldn't find any good OS tools like it

Wish it was OS though since there's a few things I'd like to add, eg letting you compare the image by hovering over it

Is there something like this that works with webms?

fdupes

Haven't seen anything like it, you could probably hack something up to make contact sheets (pic related) from your webms and compare those tho

Sauce

I assume you're either using btrfs, in which case lmao, or zfs+dedup, in which case lmao

mless.com/1075BE3

I just leave dupes all over the place because I'm fucking terrible at categorizing stuff and will never find it where it's supposed to be

rmlint

find -exec md5sum {} \+ | sort -k1 | uniq -w32 -D
find files and execute md5sum, sort by the md5s, and then find duplicate md5s and print them. this takes care of strictly identical files

for visually similar images i use an image-based fingerprinting algorithm in place of md5, and i test for "close" fingerprints instead of perfect matches

That would take forever.

Either:

1- parallelize it to compute many md5sums at the same time
2- only compare md5sums for files of same size (so sort by size first)

3- use an existing tool that already does all of this better

common sense

video-comparer.com/

only good solution for video that I have seen, I paid for it.

Well there's btrfs which is great

I like meld.. it's OK

duff

>Well there's btrfs which is great
top meme

What zfs

Try vistanita duplicate finder.
It's very easy to use and it's full of features.
It also has different modes for finding bit-by-bit duplicates, pictures of different res or quality, duplicated music, etc.
You can also set many different parameters to finetune your research.
I use it almost every day, I love it.
If anyone wants, I can upload my cracked copy in an hour or so, when I'll be on the pc.

What english

I know my porn very well. I don't download it twice :^) Only solo/lesbian, btw

check md5

Find locate sort uniq

find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

fdupes works pretty good for Sup Forums downloaded jpgs. (filename and md5sum)

>MD5
mscs.dal.ca/~selinger/md5collision/

fdupes -rdS /path/to/folder
Finds duplicate files, shows you the filenames + size, and asks which to keep. I use it to clean my reaction images folder every once in a while.

Anyone know tools which can look for duplicates inside zip files too? I tend to archive stuff and it'd be a pain extracting everything just to check for duplicates inside.

Nothing, I don't give a shit.

Looking for dupe reaction faces, not verifying nuke missile launch codes.

You gonna share with the class here user?

Best post.

literally the developer sperging

i'd make a joke about how you're probably desperate for donations but you're so autistic you don't even accept them LOL!

>windows
>most comprehensive docs

Before the thread 404s, here it is if anyone wants it:
www58.zippyshare.com/v/j2xkenLO/file.html

nice malware

Actually yeah, it could have malware kek, as I've probably downloaded it from a sketchy source.
Inside that zip there are the .rar archive as I've downloaded them, so you can scan them if you want.
If there's malware, I certainly wasn't the one wo put it there.

sure that sounds very believable

Upload the file on VirusTotal and check the date of the first time it has been scanned.

virustotal.com/en/file/15f15086de102939941f2b0e784309f434810df6afe75ab2665ed0020bffd117/analysis/

This obviously means it hasn't been touched since.

Fuck, this was meant for

>detection ratio: 1/52
lol

Also,
1. virus scanners are trivial to defeat. You can take any malware and slightly modify it to bypass the check
2. just because you wrote your epik malware a few years back doesn't somehow make it stop being malware

>Implying that's not the obvious false positive from the cracked exe
>Implying malware made (and spread around the Internet) 6+ years ago wouldn't be immediately detected now
>Implying I give a shit if you install it or not
>Implying I'd stay here putting all this effort to make a couple of tech-savvy people install my old malware, when I could easily put it somewhere else and let hundreds of tech-illiterates download it without having to defend myself.

I'm just trying to be helpful. If you don't want my version, you can find it elsewhere, or not download it at all for all I give a shit.

Also the 3.9.5 actually seems to be risky, so don't install that.
I just had it in my folder and uploaded it without checking.

Bumps

>Implying that's not the obvious false positive from the cracked exe
Yes, yes, I'm sure your crack is a “false positive” ;-)

>steps to install
>0: turn off ur antivirus
>1: run exe
>2: click “ok” on all driver certificate warnings that show up
>3: IGNORE any antivirus messages, they are a FALSE POSITIVE
>tested 100% clean cracked by SKIDROW

Wut?
Where did you read that?
It's not in either of the folders.

...

SimilarImages for images it´s so far the only one that i could find that is able to compared over 50 gigs of images and find those that are similar

Wincuccs REKT

Do I need to point out how I know you are not a dev with that shitpost?

I use a filesystem with native dedup since I'm not a pleb.
>what are inodes

and what filesystem might that be, memelord?

Mk I Eyeball

nirsoft.net/utils/search_my_files.html

zfs on my storage server running freebsd.
Wouldn't want to be running the hack linux port.

>he has zfs dedup turned on
hahaha oh wow

enjoy your 0.X% storage gains in exchange for nuking your ARC size

I've got 64GiB of RAM, and a 256GB SSD for cache. Why don't you?

Nice , just tested it

IT JST WRRRKS

It is sometimes hard to trust links on /g , I do not want to become bitcoin miner for someone else

but this time it is safe

S-Sorry, my L2ARC is only 128 GB because I had to sacrifice one of my SSDs for a barebones build

Still, I have 64 GiB of RAM as well and I have dedup turned off, because the gain is not worth the cost no matter how much spare RAM you have.

ps. show us your `zpool get dedupratio`