ITT thread we discuss how to drive thrown the click-rates for major news sites aka sites of disinformation
for example: is it impossible to organize a community-driven newssite? something needs to be done, I can't stand these shills blowing government's dick.
maybe something that auto-pirates the articles of newssites and aggregates them for us? anything that will hurt ad revenue.
Other urls found in this thread:
m.youtube.com
npmjs.com
npmjs.com
youtu.be
en.wikipedia.org
github.com
ampproject.org
zdnet.com
peopleinformer.com
en.wikipedia.org
twitter.com
>auto-pirates
Scrapes?
>Scrapes
never heard that term in this context. what are you talking about?
like drudge but instead of linking to articles it links to archives?
ddos them all the time.
Napster for news.
Make it a browser extension.
Make it break through paywalls.
Make it better to use than actual newssites.
Give it corss site features, auto comments, social media integration.
Make it distributed, decentralised, redundant, change user behaviour.
Make it good enough and you could kill online news in 36 months.
web scraping
look it up
i tried this...
if there is enough support i could fire up the servers again...
- scraped news from 2000+ news sites. (scrape the html not shitty rss)
- had a huge digital pipeline setup on the incoming news todo processing, (things like pulling company names out, or linking to people on wiki, AI for content curation and topic analysis, could put coincidence detector on everything)
- aggregated on website, ipad, iphone and android app.
wrote about 200k lines of code to run:
- digital news desk
- scraper
- aggregation pipeline
- website
- mobile app
it was called bullet news, google it...
pulled the app earlier this year, not enough users, media getting pissy, demanding money for content...
i occasionally use the scraper to catch big stories as they come in.
seems a waste to have it all sitting there not being used.
:(
what you think user?
i mean, its ready to go...would you support it?
call it "pol news" or something
biggest issue is legal not technical.
either:
- you pay for content
- media come after you
that simple...
MMS have Stupidly good (((layers)))
i wouldn't commit to any course of action. this is somethign that would have to be carefully planned and executed. and you would have to get as many people on board as possible.
the first step is building sufficient awareness that the MSM is full of sh*t and that their click rates (=ad revenue) must be hit. i just don't want to have to go spiegel or welt online anymore, knowing that every page hit brings them money. I want to be informed without them earning a cent.
when we have broad enough awareness/consensus, you have enough brain power to devise a workable solution.
you said it yourself: IF there is enough support. you need to secure this support long in advance. you need a broad base of supporters first.
interesting, how would you deal with the msm getting asshurt about it though
how do people who pirate movies and music do it? why can you pirate a movie but not an online available news article? the one issue I see is reliability. readers need to know you're not changing the content to suit your own agenda.
That sounds cool, but don't associate it with here, draws unnecessary attention from shitposting normies. Unless you forego the coincidence detector because anymore normie is going go full >muh antisemitism, it will just delegitimise it, you know?
you can make it decentralized like streaming websites. but then I think the biggest issue is trust: uploaders might try to change content to suit their own agenda.
to avoid that you'd need editors exposed to enormous legal risk.
you basically have to pay them for content.
its NOT cheep.
and in a dying market, they know MSM is on the way out, they are getting SUPER greedy...
- reddit gets away with it because its just a title
- facebook gets away with it because MSM doesn't have a choice, they follow the users, and fb has Billions.
Every billion pound tech company is now in news:
- twitter
- facebook
- spotify
- apple
All just add to the problem tho, they all copy/steal "news" from MSM, all the MSM just copy news off the wire or each other...
I was at a news conference once, in front of me was a jurno for al j... i watched him grad a story from the guardian and copy paste into their CMS verbatim...
explains who MSM is so full of shit...
the issue isn't with ripping the news.
thats super simple.
can write a bot to scrape news sites VERY fast, and most don't mind...
The issues come when you try to publish them.
You would HAVE to have it as a non-profit, not only so people trust you... but also to get around the legal from their side...
But if its a non-profit, who pays the devs?
There are 1000's of man hours required to make something work well...
Bumping for interest
Fuck the media
it'd have to be decentralized and verifiable. something like blockchain technology. It'd also have to have stringent anti bot measures or people will bot the living fuck out of it for profit.
>The issues come when you try to publish them.
Please explain. The internet is already full of streaming sites with illegal content.
also you'll realize that it won't be just conservative news. it will be news of any kind, including commie, nazi, jihadi, and everything else under the sun.
yeah 110% agree...
the post processing was more for:
- AI to guess topic (DBN works well)
- Natural language processing
- Pulling out Info and linking to relevant websites
- people
- places
- companies
i wanted to have it setup so, for example:
- a business man could invest in a random company, say "yandex"
- they could drop on site and setup complex alerts like
"any story mentions yandex and has negative sentiment" send me sms, email or webhook...
- business man can relax easy, knowing his money is safe.
i did make a kiddy script thing for node.js
basically turns news sites into a json stream.
When they publish a new article, you get json or article + augmented data...
pic related.
they do their best to shut them down.
once you are big enough to cause them lost revenue, they come after you...
settlements start in the millions.
Feedly, gets away with it because the user opts into grabbing the news from particular sites.
so its more a "tool" than a news site...
No reason something similar couldn't be setup.
MSM is about 10-15 years behind on the tech side of things...
Another reason why facebook and other giants have got into it...
The whole industry still relies on RSS, XML or other nasty as shit things....
Even if you just had an aggregation engine that spat out json or a rabbitmq queue it would be the worlds best src for news...
Then let people build whatever they want on top...
this makes no sense. streaming sites have cost the entertainment industry huge amounts of money. i mean today bands make their revenue mostly by doing life shows (because of all the piracy).
i guess people are just fine with clicking articles on msm sites. the funny thing is that these media outlets are already suffering, laying off employees. it doesnt take all that much to further hurt them.
it needs to look nice or people won't use it.
Dont use them.
Dont link to them.
Insult anyone who links to them.
Redpill your anyone who meet.
Thats all you can do, anything else gives them attention.
agreed.
One of the biggest issues with MSM is the current loss of revenue...
Even someone like the BBC is firing their jurno's and rely heavily on News agencies (Reuters, AP, AFP) to tell them the news that day...
All MSM sing the same biased song because they are all copy + pasta from the same place...
>web scraping
illegal
says the guy on Sup Forums...
it didn't look too bad.
ignore the shitty caption.
it is if your using it for profit.
read t+c's of any news desk.
> Natural language processing
NLP can't filter the bullshit. We need Neural Network.
Website related OP
We can make a good alternative media source happen right now just donate a dollar a month through patreon
www.ancreport.com
Video related for newfags not familiar with his work.
He exposed what was in the 28 pages in it's near entirety back in November of last year due to his research
He also goes over some of the 1800+ pages of the same kind of FBI reports the Senate referenced for the Saudi role to expose the much more massive Israeli role in 9/11.
filtering is a HUGE issue...
from 2000 news src's, it was picking up 100k+ articles a day...
at BEST people read about 50-100...
NLP was to fix their shitty formatting / titles...
and also allow a WAY nicer alerting system.
alert for "bomb" would also pickup "bomber", "bombing"... NOT using stupid regexp.
also allows nice features needed by any news desk:
- profanity filters etc...
i used these package + others:
npmjs.com
npmjs.com
> 2016
> wordpress
NO.
page load took 4 seconds...
> aint got time for that shit
needs to be full page load in less than 200ms.
But I would like to get uptodate news (with pictures).
it needs to have a bigger reach/appeal than Sup Forums to make a dent in MSM revenue
you might need a reddit style up/down-voting mechanism
By confirmation bias.
>someone goes "Is this real"
>Post "yes it's real there are several sources for this"
This is why normies cannot dig to our level. Normie confirmation bias forces the shit pages to stay in public spectrum.. even if it's completely wrong.
This is final boss of this election campaign.
Hence needing money
Content is more important than load times and this content is needed to be seen by millions.
It is succinct to the point reports on things actually important and most importantly truthful
youtu.be
Offer to work for him then and improve the website
IF enough people are interested.
Seems a few on here are already.
i'll put together a few ideas i had on MSM major issues and ways i thought to solve them...
1. comment sections are kill
MSM has shutdown on their comment sections soooo much they are now less than useless, and only go to support their echo chambers.
Solution:
- opt-in group comments
imagine if you went to bbc comments or something, but it only showed you comments by your "groups" a list of tags that you opt into...
By opting in, it means the users chooses to see "#blm" comments, or "#pol" comments...
If you don't know the tag, you dont see that groups comments...
- private comments
you can go to news article,
highlight sections, add annotations + comments...
and save.
then url turns from:
"bbc.co.uk/news/something-shit-happened"
to
"bbc.co.uk/news/something-shit-happened#comment-345345" or something...
so you can then share the url, and anyone who clicks above link will see article + your comments....
Then they could comment further, and url changes again...
So, if you don't have the link, you don't see comment stream...
nice idea.
also we will need some kind of shill deleting mechanism
if you're serious about it i suggest you keep making threads about this, brainstorming shit. this will create the necessary publicity on Sup Forums to get more people on-board, build support on the imageboard, etc.
i was thinking about how Sup Forums works...
you can't 'delete' easily...
it either needs super amazing devs, or real people to edit.... both cost money.
you could have mods, but then its just reddit cluster fuck again...
does anyone know, or have links to how Sup Forums manages content...?
it would be foolish to not take inspiration from here...
community driven is possibly best option, but depends on the quality of the community...
Hey bong,
In reference to your earlier post about needing 1000s of devs and being exposed to liability...Separate entities. The tool can be run like any open source non-profit project. The administration of the tool would be a different entity.
MSM comes after the tool
>we just make software dude
MSM comes after admin
>we just make sure the tools intertubes run smoothly
okay... i'll keep on it...
ANONS WAKE UP...!!
=================
The MSM is the key to our suffering, as we all know.
Why are we so happy to expect so little quality from OUR media?
WE MUST DEMAND MORE...!!!
Facts:
- MSM is dying
- MSM is shit with tech
- MSM is NOT FREE
- MSM is biased
- MSM copy + pasta every other MSM, (feedback loop of death!!)
WE MUST STOP THIS...!!
We have 2 choices, but must pursue both..
1. Hurt the MSM (ad revenue), to show we are not happy with their (((news))), in the hope they will change...(protip: they wont)
2. Build our own MSM to replace them.
>inb4: newfag thinks he can take (((them))) on
>inb4: thinks Sup Forums isn't full of NEETs
So, step 1...
I'm compiling a list of MSM we use...
Doesn't matter if you like or hate them, we need them all...
Post MSM urls below like so:
> bbc.co.uk
> theguardian.co.uk
> news.sky.com
> rt.com
can add optional comments if needed...
All flags welcome!!
pic related
Okay, a quick overview of how Sup Forums works, and why it works:
Anonymity
No moderation
No voting system
Nobody has any skin in the game. You can't get popular here. You can't make a name for yourself. You can't attain a cult following. People actively despise anyone who even tries to attention whore.
The lack of moderation means every opinion, no matter how stupid, gets heard. There's no censorship. You aren't forced to listen and believe propaganda, and can speak out at any time to call out the propaganda.
The closest thing to a vote that Sup Forums has, is a reply. The posts that get the most replies are usually those that are most controversial or funny, replies don't necessarily mean a post is correct, but they do tend to indicate the post is worth reading, or such utter garbage that it's worth spending time letting them know.
The single biggest issue with all this is subversion by bots/shills who market/propagandize.
In the context of news reporting, there are some very important things people need/want:
1. Unbiased reporting. Bias introduces itself easily, just in the way a headline is worded.
2. Informative discussion. Insightful comments from other people relating the event to other events.
3. Simple enough to understand
The ideal news is 100% free of bias with simple but informative explanations of how it relates to everything else and why, and presented in an emotionally appealing way that engages the mind without overwhelming complexity, and ultimately reveals the actual truth of what is happening.
here's the thing: you need to order news.
you clearly need headlines, even Sup Forums has stickies on happenings.
will the order be community-based or editorial staff-based? both are immensely vulnerable to bias and manipulation.
ill start:
> bbc.co.uk
> theguardian.co.uk
> news.sky.com
> rt.com
> cnn.com
> express.co.uk
> dailystar.co.uk
> enca.com
> euronews.com
> huffingtonpost.co.uk
> ibtimes.co.uk
> independent.co.uk
> latimes.com
> liverpoolecho.co.uk
> manchestereveningnews.co.uk
> metro.co.uk
> economist.com
> nytimes.com
> observer.com
> telegraph.co.uk
> thetimes.co.uk
> wsj.com
> indiatimes.com
> upi.com
> usatoday.com
> walesonline.co.uk
>You can't get popular here. You can't make a name for yourself. You can't attain a cult following.
What the fuck?
Confirmation bias disagrees. That's how Trump got support initially here.
And the kek meme.
Trump doesn't post here
Throw on a tripcode and try and get popular. I'll grab the popcorn.
>Trump doesn't post here
I didn't mean that.
The natural confirmation bias may have done this.
>Throw on a tripcode and try and get popular. I'll grab the popcorn.
heh... might try this one day
following up on this:
content and order needs to be customizable. i mainly read moderate conservative news but i also like major articles from leftist sites. dont want an echo chamber.
>might try this one day
wtf, i hate you now, faget
>spiegel.de
>welt.de
>faz.net
>focus.de
>zeit.de
>taz.de
>sueddeutsche.de
>will the order be community-based or editorial staff-based? both are immensely vulnerable to bias and manipulation.
Either of those are STILL open to bias.
Eventually a meme emerges and that becomes the bias. see
I was kidding. I know Tripfags never fucking gain popularity.
> Anonymity
Agree very important.
> No moderation
Agree very important. plebit otherwise
> No voting system
Agree very important. ditto.
> The closest thing to a vote that Sup Forums has, is a reply.
Yep, i like this too...
In effect its user time in writing the reply.
Can't fake that shit..! (excluding bots)
> The single biggest issue with all this is subversion by bots/shills who market/propagandize.
Bots aren't too tricky to solve.
Shills i would think are the biggest issue. If they are getting paid, then the whole 'time for reply' goes to shit...
> 1. Unbiased reporting. Bias introduces itself easily, just in the way a headline is worded.
This is what we all want.
I did wonder about a news desk of Sup Forumsacks after all we are everywhere already.
But... maybe we take a different tone.
The issue with the MSM is no one is calling them out on their bullshit...
What we need is a way to critique the news... And them see it...!!
> ... the way a headline is worded
This is what annoys me about comment sections at the minute.
They are a stupid text interface for what can be quite a nuanced critique of how an article is written.
2. Informative discussion. Insightful comments from other people relating the event to other events.
Hmmm.... interesting...
the community could "decorate" the news article with additional info maybe...
3. Simple enough to understand
Yep... everything has to be simple these days...
> MUH ADHD's
Okay so:
Whats your thoughts on something like this...?
Simple website with text box:
- paste in link to news article
- if the page already in system (url hash), takes you to that "article page"
somedomain.com/news/1488 (for example)
- if not, does a super quick crawl and grabs, title + meta + text + images from their website, then takes you to "article page"
- on "article page", have article nicely laid out, no ads, no bullshit...
- you can then kind of "add notes" to any part of article, maybe highlight "tanned german" or something, like annotations... can add as many as you want...
- when you "save" your notes, url changes to something like..
somedomain.com/news/1488#notes-123123
- you can then share that "note thread".
- so anyone WITH link can come back to your notes...
- they can then add further notes to you notes, or add more...
- you see any notes attached to the one your on...
- option to be public or private 'notes'...
if public, they appear in some sort of "feed" on homepage...
- articles with "best" or most notes, get to the top...
its kinda like a "marking tool" for MSM...
but we do the marking...
:/
thoughts?
>make a website for people to discuss news
>filter people and only let in real/good people
pick one user
you'd need an insulation layer between sites that publish articles/links (streaming sites) and servers that host the actual articles.
perhaps via url shorteners?
>i didnt know it was a ripped article I was linking too, I thought I was linking to the actual cnn article
What if the server gets raided?
imagine archive.is with annotations/comments...
> What if the server gets raided?
what am i doing that illegal?
how does archive.is get away with it?
:/
...
can you make users select which news sources they want to scrape, or make 'tabs' of the story and when you click on the tab you see every news story on the subject from all sources?
the first problem is that it's 3 people in this thread now. not nearly enough.
browse by topic
browse by media
browse by "custom" maybe
That's why it's not a shitfest. Should I invite some rooniggers or leafs?
>what am i doing that illegal?
Being a terrorist.
>seriously, that word is used for anything now
>how does archive.is get away with it?
They obviously distribute and change servers regularly.
Or use multiple proxies.
Or even maybe connect their internet in an infallible way (like stealing cable)
What does that mean though?
It doesn't say how their site is structured. It probably still has a server.
Re: Shilling mitigation
Admins can tag #shill onto a piece of content. Users would experience "exclude #shill" but could opt out easily. Users would have a check on the admins use of #shill by being able to set filters of their own.
Perhaps a mechanism where the community can submit content for #shill or remove content from #shill.
think along the lines of:
- global cached news archive.is
- with comments++
- no mods, no filters, no ads (pol rules)
Make a blog and go nightcrawler style, get a camera, a car, and become a fucking news outlet.
>Perhaps a mechanism where the community can submit content for #shill or remove content from #shill.
This. Also a mechanism where users can rate other users.
1. You need a place to store the cache
2. That place can be raided
However - what about using cloud servers?
>having to raid google's servers to extract the cache
I think that would hinder them.
And I don't think that would be against Google's terms.
>will Google allow it though
That's another issue here.
You ever read Snow Crash user?
Mainstream news is shit though. Why do you want free propaganda?
Unz.com
Taki's Mag
Sup Forums
Daily Stormer for you nazis
Breitbart
Zerohedge
good enough news sources right there
No, I use zerohedge but it's not enough. I do want to read what the MSM whores think/write. I just want to deprive them of their ad revenues.
I'd support it along with a dozen or so of my mates.
>Reagan went from 39 to 47 in the final stages
What the hell happened?
Classic October surprise
en.wikipedia.org
I second this. I don't know how to do any of the technical stuff but I'm more than willing to brainstorm or give opinions.
IM IN.
>Make it break through paywalls.
Ublock origin does that already at least for my local paper's website. Probably others too.
Make this the a general and use this post as the op.
Would it be possible to use torrents for this?
Ok, brain working again, not that good with code so don't get mad at me
>frontend search engine set up in a server in bumfuck nowhere
>search words,date,etc
>magic scrapes news sources and publishes torrent
>>frontend search engine set up in a server in bumfuck nowhere
I think here might be a good idea.. if you could hide your location.
You'd need wireless communication on a large scale though. Or ALOT of cabling for power and internet.
But you could definitely hide in the outback if you weren't needing such connection.
>I'm sure many people do, infact I'm certain (because I came into contact with one) that some crims are out there.
thanks for support anons...
paywalls are fucking stupid and easy to avoid.
- the 'paywall' is usually just visual...
- the article is in the html, usually for google to crawl, i use sizzle selectors to grab the correct tags...(github.com
*pic related
>torrent
needs to be web based or vast majority of people wont use it...
torrents could be useful for archive/distrobuted backups...
> hiding servers
- front end behind cloud flare, and sitting in public cloud... cant shutdown all ips.
- backend (scrapers) sitting somewhere, can proxy traffic behind 1000's of free proxies...if need to hide src for whatever reason.
but again, from legal side, unless its for profit, i think you can use...
its how google gets away with it...
more "cache" than steal...
Have a look at "Google AMP"... ampproject.org
News providers give all their content away FOR FREE to google...Why couldn't someone else use it also?
At very worse... you dont make it about news...
just make a archive.is with Sup Forums style comment section...
While we're at it I have a question that's bothering me for a long time, would it be possible to build a script that embeds Sup Forums comments inside pictures before posting?
The idea is to prevent search engines from finding threads
hmmm... interesting.
so the website looks "blank" and just full of images to google...?
you know google are ocr'ing images to extract text, have been for a while...
i'm pretty sure ever images get thrown through AI to pull out schematic "content"...
one way is just to not have an "index page", like 4chans archive...
so they can't scrape all the links.
but then article distribution is 100% community driven... via shares etc...
Yes I know about the ocr thing but then again they use captchas to solve house numbers and street names for them.
Another problem right now is that there are a couple of sites scraping entire threads from here which are then published with random usernames
Maybe use a random fractal generator to produce backgrounds? The seed for said backgrounds could be tied to the day it was posted as to prevent shilling and endless copy paste shenanigans
>its how google gets away with it...
Google was fucked big time in France and Germoney is butt-hurt as well.
>zdnet.com
They are all still siting there just because of "muh SEO", but if you're not google they will fuck you up in the court if you're scraping their content on daily basis.
You don't have idea how entitled are these faggots about their "copyrights" and "ad revenue".
Sauce: I was working 2 years for a big media company.
NON PROFIT
how does archive.is do it...?
its 80% of the way there...
just needs a filter free/mod free comment section and its perfect...
It is no different then loading the web page yourself.
Sharing what you scrap is the illegal part considering you dont have property owners permission.
This is actually a great project.
Imagine a PopcornTime for news(articles), with heavy Sup Forums bias.
Im really busy shitposting right now so get on it. Maybe we can integrate something like Pokemn Go so we can catch all the useful idiots.
>is it impossible to organize a community-driven newssite?
Check out
Anyone can register and publish legit looking articles for free.
I hate to be the pessimist! But this still does not stop them to drag you into the court and waste you years of time and piles of money, if they start seeing you as a threat.
Most online medias are forbidding scraping by their TOS and robots.txt
bump for potential
I work in telemarketing and graduate with an IT degree in december, I have a fuck ton of free time until then.
I want to help.
Here's an idea I thought of -
A voting system based on biases. The articles could be tagged with different biases, or "unbiased" should it hold up well.
Problem is there are a lot of biases
en.wikipedia.org
And on top of that, argumental fallacies should the article be of opinionated work. However, an effective voting system based solely on negative aspects of articles may be more effective than what reddit does (good / bad), and is quite nice for freelancers.
Think like anyone can anonymously (with a fun username, maybe) upload news, and readers can tag it with biases should they find some heavily displayed. That way you know what you're getting into when you click on new articles, and encourages debiasing..