I fell for the 16GB of RAM meme

>I fell for the 16GB of RAM meme
should've gone for so much more. How do I get my program to work without buying a better computer?

Other urls found in this thread:

catalog.data.gov/dataset/crimes-2001-to-present-398a4
twitter.com/NSFWRedditImage

Well, among the obvious options you could try optimizing it or writing it in a more efficient language. What kind of calculations does your program do that you run out of memory though?

training a machine learning algorithm with a 2.3GB dataset

I love 16 GB of RAM.

> he fell for the “8 threads is enough” meme

Optimize it or throw it in the trash. This is unacceptable.

>two people telling me to optimize
how tho

> just optimise this machine learning algorithm
bruh, knowing how to change and improve it is a big step up from just knowing how to use it
Not OP, just someone who ragrets going into ML for postgrad

retard

did you even try to optimize the dataset? if you have such amount of data you get fuzzy output. why not just feed rng into the maschine. wont make a difference on the output.

I did. I'm only using three out of >30 columns, so it's not the entire 2.3GB.

>ITT
>OP teaches maschine learning algorithm how to be a faggot by feeding it 2.3GB/30 rainbow flag jpeg

actually I'm teaching it how to predict the profile of a murderer based on his victim and weapon of choice plus a few other things

doubt there's much correlation anyway but I'm bored and have nothing else to do

How is it supposed to be useful when you only give it a fraction of the actual data?

do you want to make the maschine a full faggot? we already have OP

if you compile a dataset with jut such stats it cant be several GB even if you take the whole population of a country and a 100 year time scale

how are your data formated? are they even formated?

Stop using shit written in Python. That language has the most worthless memory management and garbage collection on the planet, which is just another reason why it's the worst language ever created.

I'm still in the preliminary stages of working with this data, and seeing as how I can't even get this small subset to work I may have to reconsider this project

>maschine

>quasi-English sentence which I think is saying my dataset can't be that big
lies
catalog.data.gov/dataset/crimes-2001-to-present-398a4

recommend me some ML libraries in C++ that are as simple to use as Python's and then you'll have a decent point maybe

>using python
>wondering why no memory

Shit I only have 16 threads. Do I need to upgrade?

>Using python for machine learning
>Doesn't understand why he has no memory free

>I want to do machine learning but I don't want to write anything myself
You're worse than a web developer

You add more CPUs; I'll add more RAM.
Agreed?

I've been doing machine learning for 2 days so I don't really know what I'm doing

>quasi-English sentence which I think is saying
>lies

just because they offer these data to you doesnt mean that you have the ressources to handle them. you must be special if you cant come up with that yourself.

also dont act like you know what your doing
you try to take an algo from the web and some data and think you can just somehow blend them together and get a usefull output.

your first point is completely irrelevant

your second point is completely correct; I have no idea what I'm doing and don't claim to

optimise a library?

Then you should especially not be using python, you should be learning how to build simple neural networks and develop feedback systems.

If you were a data scientist with access to a server cluster to just toss some python scripts on to run for 5 days with no concern for resource consumption, that'd be fine.
But that's obviously not the case, so do it the right way and learn something.

sudo ulimit -m unlimited

im a data scientist and everyone i know professionally in the same field uses either python or R which are high level languages

yes, it open source isn't it? get to work freetard.

here's a tip
learn about the algo you want to use and what works best with it.
then learn about your dataset and what pieces of the data are subsets you can use to get a usefull progress ratio out of your time invested. train your algo and the chunks and see if you get the result that you expect. if not you failed and need to recalibrate.

its not a magic box you need a target you aim towards or you cant interpret the data in any frame and that means you got gibbirish

only fucking academics, open source contributors, and phd students write original machine learning code in low level languages

>having a job
>having a non-autistic view of higher-level languages

see
Python is acceptable to use in some situations, but they are very far and few between

>Noobs blindy pushing retarded amounts of data in to a general purpose python neural network library
What else is new.

holy shit have you ever used scikit learn? do you know what a library is?

...

> thinking a random forest is a neural network
hehe

viruse

not really

wiruss

My friend, look into this sysctl:
vm.min_free_kbytes

And friendly reminder: Linux sucks balls. BSD slightly better. seL4 is where it's at.

Create a swapfile

>i3
>muh key combinations
kek

I want to post my desktop but desktop threads are banned, the thread.

Look up lazy evaluation, and how to write generators in Python. You will want to process the data set in manageable chunks of memory. I.e load 300 mb of data into memory, run your linear classifier or regression heuristic, save the weights matrix or whatever results, release that memory and then load in the next chunk of data into memory.

Jesus christ, do you wear a manbun too?

>machine learning
fucking tryhard memester

>not having an 18c/36t xenon CPU with 64GB of RAM

Feature extraction and dimensionality reduction.

This, buy an ssd and create a massive swap file - no regrets

I bet your shitty program is relying on a swap you don't have.