Data Analysis General

Anyone here primarily work with data? What type of shit do you do, and what tools do you use?

Personally I deal mostly with SQL Server, SSIS, Business Objects(indirect access to teradata), Excel, and some SharePoint.

Everything I've learned has been on the job and reading in my spare time.

It's pretty based because I'm automating tasks that we've always had humans do, and making it look way more professional. Management is late to the party and getting bit by the automation bug and I'm right in the middle of it for now.


Data dweebs... lets hear your story, or AMA?

Hello, I just finished reading "Head first Java" and now I'm wondering what to do next? My big dream is to immigrate to the USA, UK or Australia and become a database manager or Java programmer just like many of my friends here in India

>Anyone here primarily work with data?
Yeah, but more on the business side. I.e. creating reports, dashboards, analyzing data, doing some machine learning. Tools: R, Python, Excel, SQL, Reporting + BI tools

>automating tasks
Please tell more. Always interesting to hear what other people are doing! Did you take a look at robotic process automation?

There are certain business processes and/or tasks that end users have to do as part of their job. A simple example is putting together a report and emailing it out to people. While there are probably a million solutions to that problem, my company has none that are IT supported or free. Getting IT to do anything takes an act of congress, so that's out of the question. Essentially I do that task the hard way and pull data into my database with SSIS, do whatever transformations are necessary in SQL, encapsulate the output in HTML (again with SQL), and then throw it into a sharepoint list that kicks out the email. I also use VBA and powershell for certain things. All of it happens automatically and people are amazed I can complete these highly customized business projects and sustain them by myself. It's getting to the point where I am making whole teams within my company look like they're sitting on their hands, although that was never my intent.

The type of stuff I'm automating is more to do with business processes and not robotic type automation. Someday, that's coming though and I want to be in on the action for as long as possible.

best book to learn ms sql server/tsql?

elasticsearch

I'm learning programming for web dev, recently went from front-end to finally learning
back-end. Any good books/sources to learn about databases like MySQL ?
At the moment I'm learning NodeJS + MongoDB via courses/vids.

I work with people who work with data, but I probably shouldn't talk about it.

They're all great people though and I'd love to become a data guy myself but the odds of me transferring departments like that are extremely slim without a LOT of shmoozing. I don't technically have the right education for it.

>sharepoint

I deal with Google Analytics data, so lots of marketing and web data. Basically, I work with the botnet all day.

>Data dweebs... lets hear your story,
>In college
>Bored during the summer
>Find out that I can learn Google Analytics for free online and get certified
>Do that after work
>Get certified
>Tell my bosses (I had two separate internships at the time)
>They put me in charge of Google Analytics for their websites on top of my other duties (both of these were basic marketing gigs, I was writing content)
>Get gud over the months
>Graduate, find work as a GAfag for a marketing firm
>Analyze website, landing page, conversion rates, bounce rates, deliver reports

It's not hard data like what you Sup Forumsuys are working with, but it's fun and easy to learn.

I completed 1 year of college. If you already have your foot in the door somewhere... a lot of places care more about the work you produce and less about what piece of paper you can produce. You'd be surprised how many people that have degrees and look stellar on paper that turn out to be god awful employees. I'm lucky that at my company you can work into positions if you are good at that particular thing. Most of the inter-company transferring they never even look at your resume.

I've heard quite a few people say how amazing elasticsearch is... but even after googling I don't really get it. Can you break it down for a simpleton like me?

Hey that's interesting... what's your salary & geographical approximation?

Hard to say which is the best to learn. I have a books247 account and can pretty much read any technical book on demand. If I don't like one I go to the next....

I'm the type that I have to get my hands dirty first... I can read a book all day long but until I get my hands dirty, it doesn't really sink in. It helps me to have a basic working knowledge before I resort to the books, so what I read makes more sense.

You can download a free version of microsoft SQL Server and do quite a bit of stuff. Google the basics. "How to Create a sql server database", "how to import data to sql server". Also w3schools was also really helpful for me when I was learning to write queries....

West/Midwest, Burgerland, 62k/year + bonuses.

I could easily be making more if I picked up some more skills, which is what I'm doing right now.

You thinking of learning Google Analytics?

We data fags are the future

I know MySQL and relational concepts (normalization, indexing, querying, stored procedures, functions, triggers). Is that enough to get a job using RDBMS's in some capacity like as an analyst or something?

Well I've got three years left on my current contract so I do have time to make a name for myself. It just depends on how much they're willing to let me work on stuff that's strictly speaking outside my contract. Sadly it's a custom system so there's no certs that'll really help beyond some basic Linux stuff.

As in sealing the future of a lot of people who lost their privacy? Yeah.

>You can download a free version of microsoft SQL Server and do quite a bit of stuff. Google the basics. "How to Create a sql server database", "how to import data to sql server". Also w3schools was also really helpful for me when I was learning to write queries....

i already know that stuf, i mean more advanced shit

for example i know you should never expose your database to the users, but what to do instead i have no idea. Create a view, a report, what? How to make users connect to it and not database?

>I've heard quite a few people say how amazing elasticsearch is... but even after googling I don't really get it. Can you break it down for a simpleton like me?
It's a document-based database. It's quite frequently used for storing logs. Like the other user, I think it's good for big data stuff too because of REST API, schemaless, it's pretty fast and the visualisation tool it comes with, Kibana.

Actually in same boat as OP. Working for national company scrubbing media consumption habits.

Mainly work with Oracle SQL, Excel, SharePoint and Python to automate pretty much whatever they throw at me.

I'm kinda rattled though because they are paying me only 48k where I've literally already automated like 50% of my job.

Btw if anyone knows a useful library for manipulating Excel files with Python I'd love to hear it.

How do I get started with this shiet

Learn a scripting language.

is powershell enough?

>I've literally already automated like 50% of my job
prepare to be fired

>Btw if anyone knows a useful library for manipulating Excel files with Python I'd love to hear it.

If you look for something instead of VBA take a look at DataNitro. Otherwise I export csvs and work on them because most xlsx implementations are so terrible

I use some obscure csv2odf script for some reports (our customer is cheap and doesn't want real tools). Works ok, it's even UTF safe.

No.
Nigga if they fire me they will have to hire someone twice as expensive as me to maintain all the code I've written.
Yeah I've heard of DataNitro but I'm not down to pay for shit.

Hmm yeah I've been trying to avoid the csv route but I'm starting to think I don't have a choice.

I'm a jr data engineer.

Primarily do work with ELK stack, filebeat, puppet, and I do our tooling (primarily, since no one on the team before was focused on it and I enjoy it) now and then. We're also responsible for the MySQL side of things too, but we have two DBAs that do that on our team, me and other two guys do everything else. We're primarily responsible for getting data from the machines/vms in colo to elasticsearch and from there we get it indexed in such a way that makes it easy for everyone internally to query it.

It's used from everything to query some customer related data (for support), to getting syslog info, to network related information. What's great is even though we're part of ops we work with every department in the company because of it. To boot it's just me and the other two guys and we manage several clusters of this stuff and index quite a bit of data.

Everything we work with is either open source and centos which is pretty awesome.

ask whatever, i'm bored right now

SOMEONE PLS ANSWER

Depends on what data they'd need. Are you talking about interacting with the data from like an intermediate application for example?

For example i have c# console/gui program in which users need to see some data from some database. I know how to connect to database using Entity framework but i don't know the theory how to present the data to users without compromising the database

i guess the same question would be if i wanted the same data presented on some website

You'll probably want a read only view or something, or a user that only has specific permissions (note: not write, delete, update, etc) for that table/db. Supply that to your application so you can still pull data, just not update it. Just be as explicit as possible pretty much.

If it's hosted somewhere that's reachable on the web you'd want to make sure you can't get to it over the net, just from the machine itself, so only accessible via localhost and the like.

So, i can create a new group called Guests or Users or some shit like that and only allow them to see views?

Just recently started learning using postgresql to analyze financial time series data and I'm thinking this is pretty cool. But I think it will unfortunately always only be a hobby because I don't have the right pieces of paper to my name, or maybe I won't have to worry about that if after playing around with this for awhile I discover some edge that I could trade on my own.

Yes, from what I know that should be possible.

Here at Sup Forums, we create the databases instead of working with them.

t. inventor of the database type now in use by Google, Facebook, Twitter and about a million other companies (and sadly enough startups).

What is the difference between view and report?

yeah, your application should be limited to what it can do.

also read about sql injection

I honestly don't remember. I'm probably not the best resource for that.

Oh wait, I do remember a view is pretty much a table made from a query, don't know what a report is.

am I the only person itt who studied statistics at university?

Hello, I have much data I wish to put into a database.
How do I go about scraping the web for such information? Any good tutorials?

>mfw my workplace a 1000 men factory has
>MDE's shared OVER MOTHERFUCKING REMOTE DESKTOP as the frontend because they are pre-2000 era and they can only be used from computers running winxp
>All of our fucking inventory is stored in Excel.
>reports are stitched and sent through Outlook.
>Robbery is rampant because there isnt a proper Asset Tracking and the transfer excel logs can get lost quite fast.
I really want to help my company and become one of the top dogs through it but i dont know where to start properly, i know what needs to be done but not how without having it bite me in the ass later with huge repercussions.

I need help

databases

learn stats

You can't do anything unless management want you to. Make suggestions, you could even do a PoC on your own time if you want to learn some skills, but don't put yourself out there if they don't care.

I wrote a bash script to delete entries from Firefox SQLite databases. Can I put this on my github, is it enough to get me a job?

don't hate the player hate the game