Sunday, September 10, 2017

How To Scrape Any Website And Not Feel Bad About It

In my world, screen scraping is essential to making a living. For years, I've built applications for clients that automate tasks on major websites and, in most cases, if not all, scraping data from these sites was a big part of how the application worked. If screen scraping somehow became illegal, my business would fold. But that won't ever happen.

Periodically I read about this hilarious idea that a website has the power to somehow block scrapers. It's an understandable goal from a competitive standpoint -- after all, if you're trying to make a buck, then the last thing you want is any random joe downloading your entire website, re-branding it and taking your customers away. Including anti-scrape clauses in your terms of service makes sense, and typically websites that do will block IPs and suspend accounts after discovering the scraping. But to say they can stop the scraping altogether is ignoring how the web works in the first place.

If you can read a website in a browser, then you can scrape it, period.

The only real way to stop scraping is to not allow access to the website anymore, such as making certain sections of the website public and others requiring the creation of an official account, which is exactly what LinkedIn did -- before getting sued for it.

Now, I'm not going to say that I agree with the lawsuit. In fact, I take LinkedIn's side, considering they are not just protecting their business but protecting the privacy of their users, both of which I value. Plus, the stress that heavy scraping can put on servers can add extra costs to the business model. That could actually be grounds for a lawsuit against those doing the scraping.

However, it really doesn't matter if LinkedIn, or any website, forces users to log in in order to access the data. So long as they don't charge for the access (and LinkedIn's free tier is barely limited at all), all a programmer has to do is create a free account then work from within that account. And with all the proxy solutions nowadays, it's really not difficult at all for any tech startup, or any random joe sitting in their mom's basement, to go to town automating and scraping LinkedIn.

Web crawlers like Googlebot, Bing, and many, many others from major websites that compile data from the web are screen scrapers at the core, it's simply how a large part of the world wide web functions. Couple this with server side JS and you're looking at a vast landscape of screen scraping that is not going anywhere any time soon, no matter how hard big sites try to stop it.


Wednesday, September 6, 2017

Alt-right Bloggers Target Tech Company Tekoso Media After DMCA Takedown Request

Angry alt-right bloggers tend to make things up to support their narrative against liberals, and a recent smear campaign against New York tech company Tekoso Media is par for the course.

The most basic example of how they do this is literally creating a story and posting it a lot all over the internet. It doesn't matter if it's true or not, what matters is that it's out there. The self-described "journalists" run into a very big, but simple problem that anyone with half a brain could see from a distance if they actually thought about what they did before they acted:  when people figure out that you're lying about what you say, your credibility is completely destroyed and it takes years to recover, sometimes never recovering at all.

When it comes to fake news, alt-right vloggers and their baseless conspiracy theories dominated Youtube until Google's ad network got tired of it and stopped showing ads on their videos over the past year. This happened because advertisers decided to boycott fake news and hate speech on the Google network, which caused a significant drop in revenue for the publishers of videos containing this type of content.

The vloggers caught for copyright infringement claim that they did nothing wrong, but Tekoso Media claims it's copyright infringement takedown request was legit. The truth is that the process of a DMCA takedown request through Google is very stringent. First, they ask you to fill out a very detailed report of the infringing content on their network. Next, they ask for proof from the claimant that they are indeed the owner of the infringing content. Once verified, the content is removed from Youtube. It is assumed that Google would not follow through with taking anything down from Youtube if it didn't see the infringing content on the video somewhere, so they must be complying because they saw the infringement happening.

Many so-called "independent journalists" from the alt-right have gotten into libel lawsuits, usually against them, but sometimes attempting to turn things around and claim that other people who write about their lies and conspiracy theories are making things up. No alt-right blogger ever won a libel case, though, while more have lost or settled.

The company says they have contacted the vloggers and bloggers and that legal proceedings are in the works.


Wednesday, August 16, 2017

Bitcoin May Not Be A Bubble, And That's Good For Americans

Just about every month there is a new story of despair regarding the future of bitcoin, despite the obvious upward trend, and August has been no exception. Just today, Forbes said, "Bitcoin is worth $4000 -- Why you probably should not own one" and went on to explain that "it's a bubble, you see", just like everyone said last month and the month before that. The truth is, nobody really knows what will happen, that's why it's called speculative. But is it really speculative?

The difference between bitcoin and all currencies before its incarnation is that it can be mined. Gold, you would think, would be the most expensive currency because of this, but the fact is that metals are limited and digital currencies are infinite.

"But bitcoin is finite!"

Yeah, I know. Repeat after me: Bitcoin is not the only digital currency!

Regardless of how this so-called bitcoin bubble performs, there really is no reason to believe that digital currencies are a mere fad. In fact, there is so much more data to support the theory that bitcoin is here to stay, along with the other stronger currencies like Ethereum, Dash, and others. This is due to the fact that countries are attempting to regulate cryptocurrencies as much as they can for the simple reason that digital currencies are usually decentralized, and the idea started with bitcoin itself. Decentralized currencies means banks can control them, and that means the world economy is quickly changing to move away from centralization and putting the power of the bank into the hands of the common man.

Some people are scared shirtless of what they call "the coming global collapse" but the advances in technology and controlled civilization seem to outweigh the fears. For example, the banks are already exploring blockchain technology, the primary tech behind bitcoin and most digital currencies, as a means to transfer money between banks on a global scale at a faster rate than it currently is. This means banks are embracing the technology of cryptocurrency instead of shunning it.

Although, no one should really be surprised if the big banks have anti-cryptocurrency contingency plans if or when they can't adopt and adapt quickly enough to retain control, and that is precisely what many people fear will fuel the economic collapse.


Thursday, May 7, 2015

Slave Labor and Raising the U.S. Minimum Wage

It might take a little longer than expected, but more and more politicians are coming out of the minimum wage closet and endorsing drastic raises for both local and national minimum wage that millions of Americans get paid. The most recent being Andrew Cuomo who said in an Op-ed NYTimes piece that his wage hike will go to $9 by the end of 2015 but that he proposed $11.50 and got rejected by the Legislature.

Minimum wage was started by Franklin D. Roosevelt in 1938 as a means to end outright poverty in America, but many Americans still cannot keep up with the fast increasing cost of living in places like New York City, San Francisco, and Boston. Most minimum wage opponents will tell you that they cannot successfully run their businesses if they must pay more to their workers than they already are while supporters of the minimum wage claim (perhaps rightfully so) that without it we pave the way for slave labor. A good example of this debate was seen in the movie The Big One where Nike CEO Phil Knight claims that "Americans don't want to make shoes" when Michael Moore asks why Nike pays just $4 a day to teenage girls in Indonesia instead of paying Americans a minimum wage. Unfortunately, we don't get to hear the whole truth from Knight on the subject, but it's most likely more about turning a profit. After all, if someone told you that the cost of making 10 pairs of shoes would be a total of $5 per pair and you knew you could sell the shoes for $150, you probably wouldn't think twice about taking that deal.

Perhaps the saddest part about the debate is how slave labor has become the butt of jokes from people who typically come off as liberal, such as comedian Louis CK when he told Jay Leno a story how his daughter would eat the bubblegum flavored medicine:

You don't get to say 'ewe'! You wear clothes made by kids your age professionally!
When the realities of salve labor become so horrific and out of the control of the public, we start to feel so powerless and depressed that the only way to cope with it is to make jokes.

There are things you can do to help, however. Raising awareness is the most effective way of pushing a boycott of slave labor-backed products. Buying American-made products only causes sales to go down for the big companies like Nike and Reebok. You can also get involved in the fight against slave labor, as well as voting for minimum wage hikes by getting in touch with organizations like AntiSlavery.org, Free2Work, and Raise The Minimum Wage.