Day: August 30, 2016

How to Crawl the Web Politely with Scrapy

The first rule of web crawling is you do not harm the website. The second rule of web crawling is you do NOT harm the website. We’re supporters of the democratization of web data, but not at the expense of the website’s owners.
In this post we’re sharing a few tips for our platform and Scrapy users who want polite and considerate web crawlers.
Whether you call them spiders, crawlers, or robots, let’s work together to create a world of Baymaxs, WALL-Es, and R2-D2s rather than an apocalyptic wasteland of HAL 9000s, T-1000s, and Megatrons.

A polite crawler respects robots.txtA polite crawler never degrades a website’s performanceA polite crawler identifies its creator with contact informationA polite crawler is not a pain in the buttocks of system administrators
robots.txt
Always make sure that your crawler follows the rules defined in the website’s robots.txt file. This file is usually available at the root of a website (www.example.com/robots.txt) and

Original URL: http://feedproxy.google.com/~r/feedsapi/BwPx/~3/1wZvBmoUZww/

Original article

It’s time EU laws caught up with technology

Did you know it’s illegal to share a picture of the Eiffel Tower light display at night?
Did you know in some parts of the EU, teachers aren’t legally allowed to screen films or share teaching materials in the classroom?
Think that’s absurd? So do we. It’s time our laws caught up with our technology. Here are three things that can help fix copyright:

1. Update EU copyright law for the 21st century

Copyright can be valuable in promoting education, research, and creativity — if it’s not out of date and excessively restrictive. The EU’s current copyright laws were passed in 2001, before most of us had smartphones. We need to update and harmonise the rules to create room to tinker, create, share, and learn on the internet. Education, parody, panorama, remix, and analysis shouldn’t be unlawful.
Today, our communication, creation, and conversations are facilitated through technology. But many of our normal activities — taking

Original URL: http://feedproxy.google.com/~r/feedsapi/BwPx/~3/NvEhP7REmDw/

Original article

Using Amazon Elastic Map Reduce (EMR) with Spark and Python 3.4

As part of a recent HumanGeo effort, I was faced with the challenge of detecting patterns and anomalies in large geospatial datasets using various statistics and machine learning methods. Given the size of the datasets, the speed at which they should be processed along with other project constraints, I knew I had to develop a scalable solution that could easily be deployed to AWS. I preferred to use Apache Spark, given my personal and Humangeo’s positive experiences with it. In addition, we needed to develop a solution quickly, so naturally I turned to Python 3.4. It was already part of our tech stack and let’s be real, Python makes life easier (for the most part). Given these requirements, my quest to discover the best solution quickly led me to Amazon’s Elastic Map Reduce (EMR) service.

For those who aren’t familiar with EMR, it’s basically a scalable, Hadoop based Amazon web service

Original URL: http://feedproxy.google.com/~r/feedsapi/BwPx/~3/I9i25bRGjPo/amazon-emr-spark-python3.html

Original article

FreeNAS: Open Source Storage Operating System

Enterprise-Grade Features, Open Source, BSD Licensed
DOWNLOAD

FreeNAS is an operating system that can be installed on virtually any hardware platform to share data over a network. FreeNAS is the simplest way to create a centralized and easily accessible place for your data. Use FreeNAS with ZFS to protect, store, backup, all of your data. FreeNAS is used everywhere, for the home, small business, and the enterprise.

ZFS is an enterprise-ready open source file system, RAID controller, and volume manager with unprecedented flexibility and an uncompromising commitment to data integrity. It eliminates most, if not all of the shortcomings found in legacy file systems and hardware RAID devices. Once you go ZFS, you will never want to go back.

Web Interface

The Web Interface simplifies administrative tasks. Every aspect of a FreeNAS system can be managed from a Web Interface.

File Sharing

SMB/CIFS

Original URL: http://feedproxy.google.com/~r/feedsapi/BwPx/~3/eryIINsZxNM/

Original article

Latest Windows 10 update breaks PowerShell DSC and implicit remoting, but a fix is coming

While mandatory updates mean Windows systems are kept safe from threats, the downside is if a bad update makes it through testing it gets pushed out to everyone. This is a problem we’ve seen several times already with Windows 10, and the latest update, KB3176934, is another perfect example of this. SEE ALSO: Windows 10 Anniversary Update crashing when Amazon Kindles are connected As part of a cumulative update rolled out to users two days ago, KB3176934 is missing a .MOF file in the build and as a result has broken PowerShell DSC (Desired State Configuration). Try running a DSC… [Continue Reading]

Original URL: http://feeds.betanews.com/~r/bn/~3/oCLpfZcFtYs/

Original article

Grab your popcorn, AMC lands on Roku

AMC doesn’t always show the latest films, but it does land some big names from Hollywood, and of course it’s responsible for producing some excellent original content. It’s become a go-to channel for surfers looking for something to watch. Now the entertainment network is coming to that tiny set-top box in your living room. The channel debuts just in time for the mid-season launch of Fear the Walking Dead. If you’re into zombie apocalypses then that’s certainly for you. Roku claims “Customers who subscribe to AMC through a participating cable, satellite or telco provider can stay current with full episodes and… [Continue Reading]

Original URL: http://feeds.betanews.com/~r/bn/~3/pVNxx1EJYHQ/

Original article

Proudly powered by WordPress | Theme: Baskerville 2 by Anders Noren.

Up ↑

%d bloggers like this: