The Tor Project: Building the Next Generation of Onion Services

A few weeks ago, a small group of Tor developers got together in Montreal and worked on onion services for a full week. The event was very rewarding and we wrote this blog post to share with you how we spent our week! For the record, it was our second onion service hackfest, following the legendary Arlington Accords of July 2015.

Our main goal with this meeting was to accelerate the development of the Next Generation Onion Services project (aka proposal 224). We have been working on this project for the past several months and have made great progress. However, it’s a huge project! Because of its volume and complexity, it has been extremely helpful to meet and work together in the same physical space as we hammer out open issues, review each other’s code, and quickly make development decisions that would take days to coordinate through mailing lists.

During the hackfest, we did tons of work. Here is an incomplete list of the things we did:

  • In our previous hidden service hackfest, we started designing a system for distributed random number generation on the Tor network. A “distributed random number generator” is a system where multiple computers collaborate and generate a single random number in a way that nobody could have predicted in advance (not even themselves). Such a system will be used by next generation onion services to inject unpredictability into the system and enhance their security.

    Tor developers finished implementing the protocol several months ago, and since then we’ve been reviewing, auditing, and testing the code.

    As far as we know, a distributed random generation system like this has never been deployed before on the Internet. It’s a complex system with multiple protocol phases that involves many computers working together in perfect synergy. To give you an idea of the complexity, here are the hackfest notes of a developer suggesting a design improvement to the system:

    Complicated protocols require lots of testing! So far, onion service developers have been testing this system by creating fake small virtual Tor networks on their laptops and doing basic tests to ensure that it works as intended. However, that’s not enough to properly test such an advanced feature. To really test something like this, we need to make a Tor network that works exactly like the real Tor network. It should be a real distributed network over the Internet, and not a virtual thing that lives on a single laptop!

    And that’s exactly what we did during the Montreal hackfest! Each Tor developer set up their own Tor node and enabled the “distributed random number generation” feature. We had Tor nodes in countries all around the world, just like the real Tor network, but this was a network just for ourselves! This resulted in a “testing Tor network” with 11 nodes, all performing the random number generation protocol for a whole week.

    This allowed us to test scenarios that could make the protocol burp and fail in unpredictable ways. For example, we instructed our testing Tor nodes to abort at crucial protocol moments, and come back in the worst time possible ways, just to stress test the system. We had our nodes run ancient Tor versions, perform random chaotic behaviors, disappear and never come back, etc.

    This helped us detect various bugs and edge cases. We also confirmed that our system can survive network failures that can happen on the real Internet. All in all, it was a great educational experience! We plan to keep our testing network live, and potentially recruit more people to join it, to test even more features and edge cases!

    For what it’s worth, here is a picture of the two first historic random values that our Tor test network generated. The number “5” means that 5 Tor nodes contributed randomness in generating the final random value:

  • We also worked to improve the design of next generation onion services in other ways. We improved the clarity of the specification of proposal 224 and fixed inconsistencies and errors in the text (see latest prop224 commits).

    We designed various improvements to the onion service descriptor download logic of proposal 224 as well as ways to improve the handling of clients with skewed clocks. We also brainstormed ways we can improve the shared randomness protocol in the future.

    We discussed ways to improve the user experience of the 55-character-long onion addresses for next generation onion services (compared to the 16-character-long onion addresses used currently). While no concrete design has been specified yet, we identified the need for a checksum and version field on them. We also discussed modifications to the Tor Browser Bundle that could improve the user experience of long onion addresses.

  • We don’t plan to throw away the current onion service system just yet! When proposal 224 first gets deployed, the Tor network will be able to handle both types of onion services: the current version and the next generation version.

    For this reason, while writing the code for proposal 224, we’ve been facing the choice of whether to refactor a particular piece of code or just rewrite it completely from scratch. The Montreal hackfest allowed us to make quick decisions about these, saving tons of time we would have spent trying to decide over mailing lists and bug trackers.

All in all, we got crazy amounts of work done and we all have our hands full for months to come.

Finally, if you find these sort of hackfests exciting and you would like to host or sponsor one, don’t hesitate to get in touch with us! Contact us at and we will forward your message to the appropriate people.

Be seeing you!

Original URL:  

Original article

Creation of New AALS Section: Leveraging Technology for the Academy and the Profession

    The world of legal practice is changing.   We regularly hear about new technologies that have the potential to remake long-term approaches to practicing law.  From data analytics to self-help materials, lawyering and access to information about the law is undergoing tremendous change. The AALS Clinical Section’s Technology Committee is gathering signatures to […]

Original URL:  

Original article

Why Android apps on Chromebooks are a really, really big deal (really!)

Put on your thinking caps, my friends, ’cause it’s time to get philosophical.

Ponder me this: What constitutes an “Android device”? It’s something I’ve been mulling ever since word broke that the entire Google Play Store of Android apps would be coming to Chrome OS later this year — and it’s a question I’ll ask you to keep in mind as we take the time to think through that move and what it could mean for us as consumers.

In case you were hiding out in a bunker last week and didn’t hear, Google officially announced that it’ll soon be possible for you to download Android apps on Chromebooks and use them as if they were native programs on the platform. It’s a pivotal step in the long-rumored “merger” of Android and Chrome OS — one that further solidifies the notion that this isn’t the kind of “merger” most folks were expecting (with the exception, of course, of all you smart and stunning anthropoids who follow this column).

To read this article in full or to leave a comment, please click here

Original URL:  

Original article

How not to build a thoughtful community

How not to build a thoughtful community


These guys are doing it wrong. 

First, I don’t want their comments on my blog. I don’t know them. I don’t like that they didn’t give me a heads-up their service was coming. And I don’t like their attitude.

By being adversarial out of the gate, unless something radically changes, I won’t promote their service. I won’t incorporate what people say in their comments in my posts. I won’t read what they write. I’m very good at not looking at places I don’t want to look at. It’s an acquired skill over many years of being flamed by communities of trolls. 

Slice it however you want to do it. Be welcoming to trolls and hostile to site-creators, and you’ll get lots of trolls and you’ll be actively ignored by the people who take the risk of putting their ideas online. The other way might get you some intellect, some thinking and courage. You might help discourse. This way you’re just another annoyance I have to avert my attention from. 

Yes I know I’ve called attention to you by writing a post. Just this once. 

Original URL:  

Original article

Scaling to 100M: MySQL Is a Better NoSQL

MySQL is a better NoSQL. When considering a NoSQL use case, such as key/value storage, MySQL makes more sense in terms of performance, ease of use, and stability. MySQL is …

MySQL is a better NoSQL. When considering a NoSQL use case, such as key/value storage, MySQL makes more sense in terms of performance, ease of use, and stability. MySQL is a solid engine with lots of online material, ranging from operations and failure cases, to replication and different patterns of usage. For this reason, it has an advantage over newer NoSQL engines that are not as battle tested.

In recent years, NoSQL engines have become mainstream. Many developers look at NoSQL engines—such as MongoDB, Cassandra, Redis, or Hadoop—as their first choice for building applications, considering them a single family of products that deprecates the old SQL engines.

The choice to use a NoSQL database is often based on hype, or a wrong assumption that relational databases cannot perform as well as a NoSQL database. Operational costs, as well as other stability and maturity concerns, are often overlooked by engineers when it comes to selecting a database. For more information about the limitations and shortcomings of different NoSQL (and SQL) engines, take a look at the Jepsen series of articles from Aphyr.

This post will explain why we’ve found that using MySQL for the key/value use case is better than most of the dedicated NoSQL engines, and provide guidelines to follow when using MySQL in this way.

Wix Site Resolution

When someone clicks a link to a Wix site, his browser sends an HTTP request to a server at Wix with the site address. This happens whether the address is to a Wix premium site with a custom domain (e.g., or a free site on a subdomain of the Wix domain (e.g., That server has to resolve the requested site from the site address by performing a key/value lookup URL to a site. We denote the URL as a route for the following discussion.

The routes table is used to resolve the site address to a site object. Because sites may be exposed on multiple routes, the relation is many to one. Once the site is found, the application loads it for use. The site object itself has a complex structure that includes two lists of child objects—different services that the site utilizes. Here is a sample model of our objects, assuming a standard SQL database and a normalized schema:

MySQL is better NoSQL 1

When updating a site with the traditional normalized model, we need to use a transaction to update multiple tables to ensure we preserve data consistency. (Note that a transaction is using a DB-level lock that prevents concurrent writes—and sometimes reads—from the affected tables.) Continuing with this model, we would probably have a serial key in each table, foreign keys, and an index in the URL field in the routes table.

However, there are a number of issues with the normalized schema way of modeling:

  • Locks limit access to the table, so on a high throughput use case it may limit our performance.
  • Reading the object involves either a few SQL queries (4 in this case) or joins—with latency implications again.
  • Serial keys impose locks and again limit the write throughput.

Those issues amount to limitations in the throughput and concurrency we can get from MySQL (or any other SQL engine). Because of those shortcomings, and the fact that the use case is actually key/value, many developers opt to look for a NoSQL solution that provides better throughput and concurrency, even at the expense of stability, consistency, or availability.

At Wix we’ve found that MySQL, when used creatively as a key/value store, can do a better job compared to MySQL with a normalized data model (like the one above)—and to most NoSQL engines. Simply use MySQL as a NoSQL engine. Our existing system has scaling / throughput / concurrency / latency figures that are impressive for any NoSQL engine. Here’s some of our data:

  • An active-active-active setup across three data centers.
  • Throughput is of the order of magnitude of 200,000 RPM.
  • The routes table is of the order of magnitude of 100,000,000 records, 10GB of storage.
  • The sites table is of the order of magnitude of 100,000,000 records, 200GB of storage.
  • Read latency is 1.0-1.5 msec average (in fact, 0.2-0.3 msec in one data center).

Note that latency of around 1.0 msec is considered impressive with most key/value engines, both open source and cloud-based! And we achieve that with MySQL (considered to be the basic SQL engine).

Here is the actual schema we are using:

MySQL is better NoSQL 2

CREATE TABLE `routes` (
  `route` varchar(255) NOT NULL,
  `site_id` varchar(50) NOT NULL,
  `last_update_date` bigint NOT NULL,
  PRIMARY KEY (`key`),
  KEY (`site_id`)

CREATE TABLE `sites` (
  `site_id` varchar(50) NOT NULL,
  `owner_id` varchar(50) NOT NULL,
  `schema_version` varchar(10) NOT NULL DEFAULT '1.0',
  `site_data` text NOT NULL,
  `last_update_date` bigint NOT NULL,
  PRIMARY KEY (`site_id`)

Any field that is not used as a condition in a query has been folded into a single blob field (the site_data text field). This includes the sub-obj tables, as well as any field on the object table itself. Also notice that we are not using serial keys; instead, we are using varchar(50), which stores client-generated GUID values—more about that in the next section.

Below is the query we are using, which has high throughput and low latency:

select * from sites where site_id = (
  select site_id from routes where route = ?

It works by first performing a query on the routes table by a unique index, which should return only one result. Then we look up the site by primary key, again looking for one record. The nested query syntax ensures that we are doing only one round-trip to the database to run both SQL queries.

The result, shown above, is an average ~1 msec consistent performance, given high traffic and a high update rate. The updates are semitransactional, even without using transactions. This is because we enter the full site in one insert statement, and until we enter the routes, it will not be found in queries. So if we enter the site first, and then the routes, we are ensured to have a consistent state, even in edge cases where we may have orphan data in the sites table.

Guidelines for Using MySQL as a NoSQL Engine

Using the experience gained from the above example (and other such cases at Wix), we have crafted a short list of guidelines for using MySQL as a NoSQL engine.

The main thing to keep in mind when using MySQL as a NoSQL engine is to avoid using DB locks or complex queries.

  • Do not use transactions, which introduce locks. Instead, use applicative transactions.
  • Do not use serial keys. Serial keys introduce locks and complicate active-active configurations.
  • Use client-generated unique keys. We use GUIDs.

When designing your schema to be optimized for reads, here are some additional guidelines to follow:

  • Do not normalize.
  • Fields only exist to be indexed. If a field is not needed for an index, store it in one blob/text field (such as JSON or XML).
  • Do not use foreign keys.
  • Design your schema to enable reading a single row on query.
  • Do not perform table alter commands. Table alter commands introduce locks and downtimes. Instead, use live migrations.

When querying data:

  • Query for records by primary key or by index.
  • Do not use joins.
  • Do not use aggregations.
  • Run housekeeping queries (BI, data exploration, etc.) on a replica, not on the master database.

We intend to write another blog post with further information about live migrations and applicative transactions.


The most important takeaway from this post is that you are allowed to think differently. It is great to use MySQL as a NoSQL engine, which is not the way it was designed to work. As demonstrated in this post, an example of this is using MySQL instead of dedicated NoSQL engines that are built for key/value access. At Wix, MySQL is the engine of choice for key/value cases (among others) because it is easy to use and operate, and it’s a great ecosystem. And as a bonus, it provides latency, throughput, and concurrency metrics that match, if not surpass, most NoSQL engines.

Original URL:  

Original article

Might your next e-reader come from a Circuit City?

circuit-cityToday someone on my Facebook shared the homepage of the old Circuit City retail chain, with an announcement that it’s coming back to life under new ownership. Long-time readers may remember that Circuit City short-circuited in 2009 as an end result of making several terrible corporate decisions, not unlike Blockbuster and Borders a short while later—an ignominious end to 60 years in the electronic retail business.

A quick bit of research popped up a bunch of stories from January and February indicating that the iconic brand name has passed through several hands and is now being resurrected by a couple of retail-industry veterans. The chain will open a Dallas, Texas store in June, and  hopes to open 50 to 100 stores by next year. The plan is not to try to go big-box, like the old Circuit City, but to operate on a smaller boutique scale, selling tablets, headphones, drones, 3D printers, computer accessories, and other millennial geek fodder—not unlike Radio Shack, which has had problems of its own.

The store will reportedly feature touchscreen terminals to let shoppers browse through inventory, which is kind of interesting. In a way that, plus the boutique nature of it, puts me in mind of Amazon’s own bookstore experiments, which are effectively boutique bookstores that tie into the experience. It’s unclear exactly how these tablets will work. Will the new Circuit City simply store its inventory in a back room and bring it out when someone places a touchscreen order, using the space more efficiently than the usual stock-everything-on-the-floor retail mode?

It seems a bit odd that Circuit City would expect people to come to its store to shop on a touchscreen when they can already shop on the web from home or their tablets just fine. So maybe there’s some aspect of it I’m just not seeing yet. In any event, it’s amusing to consider the recursion and inversion of shopping for a tablet from a tablet—located in a retail store.

In any event, I bought my first Clié from a Circuit City, so I’m well acquainted with the idea of buying e-readers from such a location. But is there room in the current Amazon-dominated retail environment for a new boutique electronics store, even one with a recognizable and formerly-respected brand name—especially when the closest existing chain, Radio Shack, is on the ropes and soon expected to go the way of the original Circuit City? Well, maybe—if it can do things sufficiently differently from the way any of the current brick-and-mortar retail chains do it, and that different way works.

The post Might your next e-reader come from a Circuit City? appeared first on TeleRead News: E-books, publishing, tech and beyond.

Original URL:  

Original article

Cray Unveils Open-Source Big Data Box

Supercomputer shop Cray is branching into analytics with tailored high-end systems packing open-source number-crunching software.

Cray Tuesday unveiled the Urika-GX ticking some of the best-known technology boxes and due in the second-half of 2016.

The Urika-GX is preinstalled with OpenStack and Apache Mesos for cloud and data-center abstraction, Spark and Hadoop for big analytics.

The Cray has been pre-tested and integrated with Hortonworks’ Data Platform take on Hadoop.

Cray has also built its own graph engine software that’ll run in a version of the system called Urika-GD Graph Discovery appliance.

It claims this is 10 to 100 times faster than current graph processing systems and can run on anything from a single to thousands of processors without a hit on performance.

Also, it is capable of crunching “multi-terabyte” data sets containing billions of objects.

The software runs on Intel E5-2600 v4 processors with up to 48 nodes and 1,728 cores. There’s 35TB of PCIe SSD on-node storage and Aries high-speed interconnect from Intel that debuted with Cray’s XC30 super computer in 2013. Also onboard is 22TB of memory.

Systems are available in three configurations: 16, 32 and 48 nodes via a 42U, 19-inch rack.

Cray said the system brings the capabilities of one of its supercomputers with the convenience of an appliance with x86 hardware and open source software.

The Urika-GX systems being used by customers in life sciences, healthcare, and cyber security for “high-frequency insights.”

Original URL:  

Original article

EC2 Instance Console Screenshot

When our users move existing machine images to the cloud for use on Amazon EC2, they occasionally encounter issues with drivers, boot parameters, system configuration settings, and in-progress software updates. These issues can cause the instance to become unreachable via RDP (for Windows) or SSH (for Linux) and can be difficult to diagnose. On a traditional system, the physical console often contains log messages or other clues that can be used to identify and understand what’s going on.

In order to provide you with additional visibility into the state of your instances, we now offer the ability to generate and capture screenshots of the instance console. You can generate screenshots while the instance is running or after it has crashed.

Here’s how you generate a screenshot from the console (the instance must be using HVM virtualization):

And here’s the result:

It can also be used for Windows instances:

You can also create screenshots using the CLI (aws ec2 get-console-screenshot) or the EC2 API (GetConsoleScreenshot).

Available Now
This feature is available today in the US East (Northern Virginia), US West (Oregon), US West (Northern California), Europe (Ireland), Europe (Frankfurt), Asia Pacific (Tokyo), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), and South America (Brazil) Regions. There are no costs associated with it.



Original URL:  

Original article

E Ink announces a full color electrophoretic ePaper display

Ink® Holdings
, “E Ink” (8069.TWO), the leading innovator
of electronic ink technology in the world, today announced a
breakthrough technology with its Advanced Color ePaper (ACeP), a high
quality, full color reflective display. For the first time ever, an
electrophoretic display (EPD) can produce full color at every pixel
without the use of a color filter array (CFA). ACeP sets a new standard
for bright, reflective color achievable with EPDs. The initial target
application for ACeP will be for digital signage.

ACeP achieves a full color gamut, including all eight primary colors,
using only colored pigments. The display utilizes a single layer of
electrophoretic fluid, which is controlled using voltages compatible
with commercial TFT backplanes. The fluid can be incorporated into
either microcapsule or Microcup® structures. The richness of
the colors is achieved by having all the colored pigments in every
picture element (pixel) rather than the side-by-side pixel colors
achieved with a CFA. This eliminates the light attenuation, which can be
quite significant. Like regular E Ink ePaper, ACeP maintains the
ultra-low-power and paper-like readability under all lighting conditions.

“At its heart, E Ink is a materials and technology company,” said Frank
Ko, Chairman of E Ink Holdings. “It’s this core that provides the energy
and the foundation for the stream of products being developed at E Ink.
We expect ACeP to become the basis upon which another generation of EPD
display products can be developed.”

In developing ACeP, E Ink researchers solved the very complex problem of
how to get reflective color at every pixel in a commercially viable
structure. Other approaches have utilized stacked backplane structures
that are complex, difficult to manufacture and costly. The E Ink
approach utilizes only a single backplane. Many materials and waveform
inventions were required to independently control the position of the
multiple color pigments.

“The technical team was convinced this was achievable,” said Michael
McCreary, CTO for E Ink Corporation. “E Ink’s global R&D team has a deep
understanding and experience with electrophoretic displays. During the
years of hard work applying this experience to full color, the
breakthroughs required to achieve this milestone were numerous and
frequent. We are very proud of the team’s accomplishment and dedication
to this task.”

Multiple 20” displays with a resolution of 1600 X 2500 at 150 ppi have
been constructed. ACeP will be on display during SID’s Display Week at E
Ink’s booth (#521) at the Moscone Conference Center from Tuesday, May 24
through Thursday, May 26.

About E Ink Holdings
Founded in 1992 by Taiwan’s leading
papermaking and printing group YFY (1907.TW), E Ink Holdings Inc.
(8069.TWO), based on technology from MIT’s Media Lab, has transformed
and defined the eReader market with its ePaper technology, enabling a
new multi-billion market in less than 10 years. Its corporate philosophy
aims to deliver revolutionary products, user experiences, and
environmental benefits through advanced technology development. This
vision has led to its continuous investments in the field of ePaper
displays as well as expanding the use of its technologies into a number
of other markets and applications. Its EPD products make it the
worldwide leader for ePaper. Its FFS technologies are a standard for
high end consumer electronic products. Listed in Taiwan’s GreTai
Securities Market and the Luxembourg market, E Ink Holdings is now the
world’s largest supplier of displays to the eReader market. For
corporate information, please visit
; for EPD information, please visit

Original URL:  

Original article

Proudly powered by WordPress | Theme: Baskerville 2 by Anders Noren.

Up ↑

%d bloggers like this: