Windows Desktop Market Share Drops Below 90%

An anonymous reader quotes VentureBeat’s new article about desktop operating systems: Windows 7 is still the king, but it no longer holds the majority. Nine months after Windows 10’s release, Windows 7 has finally fallen below 50 percent market share and Windows XP has dropped into single digits. While this is good news for Microsoft, April was actually a poor month for Windows overall, which for the first time owned less than 90 percent of the market, according to the latest figures from Net Applications.

Share on Google+

Read more of this story at Slashdot.

Original URL:  

Original article

SSH for Fun and Profit

In May last year, a new attack on the Diffie Hellman algorithm was released, called
Logjam. At the time, I was working on a security team, so it was our
responsiblity to check that none of our servers would be affected. We ran through our TLS config and
decided it was safe, but also needed to check that our SSH config was too. That confused me – where
in SSH is Diffie Hellman? In fact, come to think of it, how does SSH work at all? As a fun side
project, I decided to answer that question by writing a very basic SSH client of my own.

My goal was to connect to an SSH server I owned and to run an arbitrary command, say cating a file.
It sounded easy, maybe the work of a weekend. That estimate ended up being a total lie.

I started off by finding the SSH RFC. Or, well, it turns out there are at least four of them:

That looked like way more reading than I was willing to do for this project, so I nc‘d to port 22
(the standard SSH port) on a server I controlled. It sent back the banner
SSH-2.0-OpenSSH_6.9p1 Ubuntu-2ubuntu0.1, which was the format described in the transport protocol
RFC. Great, that was where I’d start!

Banner message

I skim read the top section of the RFC, then started at section 4.2
which describes this banner message. It’s actually used to negotiate the protocol version the client
and server will be speaking: the first section (SSH-2.0) describes the SSH version, the second
(OpenSSH_6.9p1) the software version, and the last (Ubuntu-2ubuntu0.1) is an optional comment
that can be used to further identify the software.

The transport protocol doesn’t cover who sends their banner first, so you can send yours as soon
as you’ve opened a connection. You can even give this a shot yourself by running nc $HOSTNAME 22,
and replaying the banner it sends you.

Algorithm negotiation

Once you’ve told the server that you speak SSH 2.0, it sends back a pretty long blob:


There’s definitely some binary encoding going on, but also some readable ASCII strings in there. So
it was back to the RFC to figure out what all that data meant.

Skimming over some text about backwards compatibility brought me to section 6,
which describes the binary packet protocol. Decoding the message with that, and after some more
skimming, I found the key exchange algorithm negotiation section.

It turns out that SSH, like many other protocols, doesn’t actually do any cryptography of its own.
Instead, the protocol defines a series of underlying algorithms that are used to guarantee the
secrecy and authenticity of your connection. In fact, SSH negotiates separate algorithms for:

  • key exchange (how you and the server agree on a shared key, that no one else knows),
  • host key authentication (how you know you’re talking to the right server),
  • encryption, in each direction (how you stop someone eavesdropping),
  • MAC, in each direction (how you prevent someone from tampering with your messages), and
  • compression, in each direction.

It also allows negotiation of the human language each side should speak, though as far as I could
tell, this is ignored by OpenSSH.

Once we send back our own list of algorithms, the key exchange begins for real. Well, that’s what
happens if the client and server can agree on algorithms. I had a lot of trouble with the server
deciding my algorithm list was invalid, so I opened up Wireshark and sniffed the negotiation for a
standard SSH client.

Wireshark debugging client algorithm negotation

After some debugging, and a lot of packet captures, I settled on the simplest set of algorithms I
could get to work:

  • Diffie Hellman for key exchange,
  • ECDSA SHA-2 with the NIST P-256 curve for host key authentication (since I couldn’t convince the server to speak anything other than Elliptic Curve crypto for this),
  • AES-128 CTR as an encryption algorithm,
  • HMAC SHA-1 as the MAC algorithm, and
  • “none” as my compression algorithm.

Exchanging keys with Diffie Hellman

Now I was getting to the challenging part – actual key exchange! This is where the client and
server agree on a secret that’s used for the remainder of the session. I’d decided to use Diffie
Hellman, both because it was what got this whole thing started, and because it was one of the only
algorithms my server supported that was documented in the original RFC.

Diffie Hellman’s security is derived from the hardness of the discrete logarithm problem.
Essentially, if you raise one number to the power of another (mod a third), then given the base and
the result, it’s very difficult to find the exponent. In this particular case, I’d decided to use
Diffie Hellman group 14, which is simply a
particular set of base, mod, and field size numbers. Implementing this in Python is pretty easy –
you generate a random number, run the pow function with the paramaters given, and you have a
shared key.

# Diffie Hellman key exchange (group 14)
# g is the base, p is the mod, and q is the field size
g = 2
q = 2 ** 2048

x = random.SystemRandom().randint(2, q - 1)
e = pow(g, x, p)

# Send e to the server
# Receive f from the server

shared_key = pow(f, x, p)

Verifying the server’s identity

Once I’d generated a shared key, I needed to make sure I was actually talking to the right server,
not an imposter. In SSH, that’s done by having the server sign all data sent so far in the
connection with its private key, and having the client verify that with a known public key.

Unfortunately for me, the server was running a fairly new version of OpenSSH, which meant it would
only verify its identity using ECDSA. I don’t fully understand elliptic curve cryptography, but
helpfully Python has an ECDSA module that could do the hard
work for me. Unhelpfully, it takes signatures in a different, and as far as I can tell undocumented,
key format.

With some guesswork, I discovered that this format seemed to be just the raw bytes of the r and
s, the two components that make up an ECDSA key. Unfortunately, these both needed to be 32 bytes
long, and for some reason, I’d occasionally find that they had an extra byte! This meant that
signature verification would only work around one in every four times.

A closer reading of the spec proved helpful here, in particular of RFC 4251,
the “overall architecture” RFC that I’d skipped earlier. r and s were being sent as mpints, or
“multiple precision integers”. I’d assumed that these were unsigned integers in network byte order
(that is, big endian), however they’re actually encoded using two’s complement. This means that the
first bit of the number determines the sign, so to encode a 32 bit unsigned integer with its high
bit set, you need to add a leading 0x00 byte.

def twos_complement(byte_array):
  byte_array = [bitflip_byte(b) for b in byte_array]

  i = 0
  while byte_array[i] == 'xff':
    byte_array[i] = 'x00'
    i += 1
  byte_array[i] = struct.pack('B', struct.unpack('B', byte_array[i])[0] + 1)

  return byte_array

def bitflip_byte(data):
  return struct.pack('B', (~struct.unpack('B', data)[0]) % 0x100)

Implementing two’s complement in Python was more involved than it probably should have been, but
once I was parsing mpints correctly, I could reliably verify signatures! These were being verified
against the public key the server presents, but I still needed to check that the public key really
was for that server.

The SSH spec is intentionally lenient here – all current key distribution systems have substantial
drawbacks, so the default is to allow users to accept arbitrary public keys the first time they
connect, and then warn them if these keys change. That’s the “Are you sure you want to continue
connecting (yes/no)?
” you see whenever you try to SSH to a new server!

I went down a bit of a rabbit hole here, duplicating the behavior of my local OpenSSH client: adding
new host keys to ~/.ssh/known_hosts file, and checking for existing keys there. However, with that
done, key exchange was over, and we were ready to move on to actually sending data!

Encrypting and authenticating messages

From here on in, all the messages sent between the client and server were encrypted and
authenticated. That rendered Wireshark useless for debugging, since it works by intercepting the

This was a problem, because my initial packets to the server were met with immediate disconnects,
and I’d now lost my main means of debugging. I banged my head against the wall for a while, then at
the suggestion of a friend, decided to turn the server’s OpenSSH log verbosity way up. I bumped the
LogLevel in /etc/ssh/sshd_config to DEBUG3, and suddenly I was getting helpful error

The bug turned out to be minor (a missing value for key derivation), however this led me to
the most fun part of the SSH spec, the hard-coded alphabet! As already mentioned, SSH relies on a
number of other cryptographic primitives, each of which needs its own separate key. In fact, SSH
needs six separate pseudo-random values, each of which it derives from the originally negotiated
shared key.

SSH uses a well-known property of cryptographic hash functions, called the avalanche effect, to
derive these keys. This property says that small changes in input value lead to large changes in
output value. By adding a single extra byte to the shared key, we can generate a completely
different key. In the case of SSH, the spec actually hard-codes the ASCII values “A” through “F” for
this. Nothing is wrong with that from a security perspective, but it’s adorable to know that those
values exist in every SSH connection you make:

Initial IV client to server: HASH(K || H || "A" || session_id)
  (Here K is encoded as mpint and "A" as byte and session_id as raw
  data.  "A" means the single character A, ASCII 65).

User authentication

With encryption and authentication going, I now had a working transport layer to start building on!

Like the OSI network model, SSH is also layered – inside the basic transport protocol, which
guarantees confidentiality and authenticity, we can run other, more complex protocols. The first one
of these that we need to run in order to do anything is the “ssh-userauth” protocol. This
authenticates the client to the server, and it’s the reason why you pass a username as part of
SSHing to a server.

There are several ways to authenticate an SSH user, some of which you’ve likely used before:
password, public key, and host-based authentication are all supported. For my client, I decided to
go with public key authentication, since that’s what I use day to day. To do this, I needed to sign
a message with the private key corresponding to a public key that I’d granted access to the server.
In practice, this meant cracking open my ~/.ssh/id_rsa file! Or, trying to – my private key is
stored in a password-protected file, and the version of PyCrypto I was using didn’t seem to be able
to parse the format.

I generated a new key pair (this time stored in plaintext) added the public half as an authorized
key to the server I was using, and then tried authenticating. No luck. I skimmed through the
PyCrypto RSA encrypt documentation a few times before noticing the very obvious Attention banner that said I shouldn’t be calling this function
unless I really knew what I was doing. It turns out that so-called “textbook RSA” has a number of
undesirable properties,
and that you nearly always want to pad your data. With the right padding in place, I got an
SSH_MSG_USERAUTH_SUCCESS message, and we were away!

Getting a shell

Now that I’d authenticated, there was only one step to go, launching a shell! Like all things
SSH, this was a little more complicated than it first sounded. SSH uses a concept called
“channels” – essentially, it’s possible to run multiple logical connections over the one transport
connection. This means you could, for example, run an SSH shell, forward X over SSH, and forward a
port over SSH, all with the same connection.

So, to get to a shell, the first thing I needed to do was to open a channel. Since I’d be
dynamically interacting with the channel, I needed to open an interactive session, then pivot this
into a shell. Initially, I’d open a channel with the “session” type, then I’d request that a
particular program be launched on the server-side. In this case, I requested a shell, but you can
also run arbitrary commands on the server, pass environment variables, or
any number of other fun things.

With a shell launched server-side, I finally got back the message I’d been hoping for – the Ubuntu
welcome banner! I was surprised to see that a prompt wasn’t being sent over the wire, but wrote a
simple read-eval-print loop that let me run arbitrary commands on my remote server. I successfully
cated a file, and could also use pipes to create my own on the server. Mission accomplished!

Finishing up

I’ve refactored my code a little to make it easier to read, but you can find my client, along with
inline documentation, here.

My “weekend” project ended up taking more like weeks, but I learned a ton about how SSH worked in
the process! I hadn’t realized that the protocol was as layered as it actually was, that client
authentication happened completely separately to host authentication, or that it was possible to
multiplex connections. Not to mention, I finally figured out where the Diffie Hellman was!

Original URL:  

Original article

Feinstein-Burr: The Bill That Bans Your Browser

Last week, I criticized the confused rhetorical framework that the Feinstein-Burr encryption backdoor proposal tries to impose on the ongoing Crypto Wars 2.0 debate. In this post, I want to try to explain why technical experts have so overwhelmingly and vehemently condemned the substance of the proposal.

The first thing to note is how extraordinarily sweeping the bill is in scope. Its mandate applies to:

device manufacturers, software manufacturers, electronic communication services, remote communication services, providers of wire or electronic communication services, providers of remote communication services, or any person who provides a product or method to facilitate a communication or to process or store data. [emphasis added]

Any of these “covered entities,” upon receipt of a court order, must be able to either provide the government with the unencrypted “plaintext” of any data encrypted by their product or service, or provide “technical assistance” sufficient to allow the government to retrieve that plaintext or otherwise accomplish the purpose of the court order. Penalties aren’t specified, leaving judges with the implicit discretion to slap non-compliant providers and developers with contempt of court. Moreover, “distributors of software licenses” — app stores and other software repositories — are obligated to ensure that all the software they host is capable of complying with such orders.

Some types of encrypted communications services either already comply or could comply in some reasonably obvious way with these requirements. Others, not so much. Because of the incredible breadth of the proposal, it’s not possible to detail in a blog post all the varied challenges such a mandate would present to diverse types of software. But let’s begin by considering one type of software everyone reading this post uses daily: Your Web browser. To the best of my knowledge, every modern Web browser is non-compliant with Feinstein-Burr, and would have to be pulled from every app store in US jurisdiction if the bill were to become law. Let’s explore why.

While ordinary users probably don’t think of their Web browser as a piece of encryption software, odds are you rely on your browser to engage in encrypted communications nearly every day. Whenever you connect to a Web-based e-mail provider like Gmail, or log into an online banking account, or provide your credit card information to an e-commerce site, you’re opening a securely encrypted HTTPS session using a global standard protocol known as Transport Layer Security or TLS (often still referred to as SSL or Secure Sockets Layer, an earlier version of the protocol). Even sites that don’t traffic in obviously sensitive data are increasingly adopting HTTPS as a default (for example, your visit to this post if you’re reading it at Just Security is HTTPS by default). Any time you see a little padlock icon next to the address bar on your browser, you are using HTTPS encryption.

This is absolutely essential for two big reasons. First, people routinely connect to the Web via WiFi access points they do’t control — such as a hotel, coffee shop, or airport. Without encryption, an unscrupulous employee of one of these businesses — or a hacker, or anyone who sets up a phony “LocalCafeWiFi” hotspot to snare the unwary — could easily vacuum up people’s sensitive data. Second, the Internet is a packet-switched network that operates very differently from traditional centralized phone networks. That means even when you’re connecting to the Internet from a trusted access point, like your home or office, your data is passed like a relay race baton across perhaps dozens of different networks you don’t control between your computer and the destination. (You can use a program called Traceroute to see all the intermediary points your data passes through on the way from your computer to any given website.) Without encryption, you’d have to trust this plethora of unknown intermediaries, foreign and domestic, not only to refrain from snarfing up your sensitive data, but to secure their systems against hackers looking to snarf up your data. Which, needless to say, is impossible: You’d be a fool to undertake a commercial transaction or send a private message under those circumstances. So it’s no exaggeration to say that the Internet as we now know it, with the spectacular variety of services it supports, simply wouldn’t be possible without the security provided by cryptographic protocols like TLS.

So how does TLS work? If you’re a masochist, you can wade through the the technical guidelines published by the National Institute of Standards & Technology, but here’s a somewhat oversimplified version. Bear with me — it’s necessary to wade into the weeds a bit here to understand exactly why a Feinstein-Burr style mandate is so untenable. Diffie-Hellman_Key_Exchange

When you open a secure HTTPS session with a Web server — which may, of course, be abroad and so beyond the jurisdiction of US courts — your Web browser authenticates the identity of the server, agrees on a specific set of cryptographic algorithms supported by both your browser and the server, and then engages in a “handshake” process to negotiate a shared set of cryptographic keys for the session. One of the most common handshake methods is a bit of mathematical sorcery called Diffie-Hellman key exchange. This allows your computer and the Web server to agree on a shared secret that even an eavesdropper monitoring the entire handshake process would be unable to determine, which is used to derive the ephemeral cryptographic session keys that encrypt the subsequent communications between the machines. (Click the link above for the gory mathematical details, or see the image on the right for a liberal-arts-major-friendly allegorical version.)

A few features of this process are worth teasing out. One is that, properly configured implementations of TLS (depending on the exact algorithms agreed upon) give you a property known as “forward secrecy”: Because a unique, unpredictable, and ephemeral key is generated for each session, old communications remain securely encrypted even if a server’s long-term cryptographic keys — which remain the same over longer periods for purposes like verifying the server’s identity — are later compromised. In economic terms, that means the “return on investment” for an attacker who manages to worm their way into a server is limited: They might be able to read the communications that occur while they remain in the system undiscovered, but they don’t then get to retroactively unlock any historical communications they’ve previously vacuumed up. This both mitigates the the downside consequences of a successful attack and, perhaps more critically, makes it less rational for sophisticated attackers to expend extraordinary resources on compromising any given set of keys. A recent paper by a who’s-who of security experts, incidentally, pointed to forward secrecy as a feature that is both increasingly important to make standard in an escalating threat environment and particularly difficult to square with a government backdoor mandate.

Some of the reasons for that become clear when we consider another rather obvious feature of how TLS functions: The developer of your browser has nothing to do with the process after the software is released. When I log into an e-commerce site using Firefox, Mozilla plays no role in the transaction. The seed numbers used to negotiate session keys for each TLS-encrypted communication are generated randomly on my computer, and the traffic between my computer and the server isn’t routed through any network or system controlled by Mozilla. A user anywhere in the world with a copy of Firefox installed can use it to make secure connections without ever having anything further to do with Mozilla. And TLS isn’t designed this way because of Edward Snowden or as some insidious effort to make it impossible to execute search warrants. It’s designed that way because a lot of very bright people determined it was the best way to do secure communications between enormous numbers of arbitrary endpoints on a global packet-switched network.

There are any number of ways a government agency might be able to get the contents of a targeted user’s TLS-encrypted HTTPS communications. The easiest is simply to demand them from the server side — but that will only work in cases where the server is subject to US jurisdiction (or that of an ally willing to pass on the data), and the server may not itself log everything the government wants. Advanced intelligence agencies may be able to mount various types of “active” attacks that involve interposing themselves into the communication in realtime, but developers are constantly striving to make this more difficult, to prevent criminal hackers from attempting the same trick — and while NSA may be able to manage this sort of thing, they’re understandably reluctant to share their methods with local law enforcement agencies. In any event, those “active” attacks are no help if you’re trying to decrypt intercepted HTTPS traffic after the fact.

Now an obvious, if inconvenient, question arises. Suppose a law enforcement agency comes to Mozilla or Apple or Google and says: We intercepted a bunch of traffic from a suspect who uses your browser, but it turns out a lot of it is encrypted, and we want you to decipher it for us. What happens? Well, as ought to be clear from the description above, they simply can’t — nor can any other modern browser developer. They’re not party to the communications, and they don’t have the cryptographic keys the browser generated for any session. Which means that under Feinstein-Burr, no modern Web browsers can be hosted by an app store (or other distributor of software licenses), at least in their current forms.

How, then, should developers redesign all these Web browsers to comply with the law? The text of Feinstein-Burr doesn’t say: Nerd harder, clever nerds! Love will find a way! You’ll figure something out! But as soon as you start thinking about classes of possible “solutions,” it becomes clear there are pretty huge problems with all of them.

One approach would be to make the key material generated by the browser not-so-random: Have the session keys generated by a process that looks random, but is predictable given some piece of secret information known to the developer. The problem with this ought to be obvious: The developer now has to safeguard what is effectively a skeleton key to every “secure” Internet communication carried out via their software. Because the value of such a master key would be truly staggering — effectively the sum of the value of all the interceptable information transmitted via that software — every criminal organization and intelligence agency on the planet is going to have an enormous incentive to whatever resources are needed to either steal or independently derive that information.

Another approach might be to redesign the browser so that the developer (or some other designated entity) becomes an effective intermediary to every HTTPS session, keeping a repository of billions of keys just in case law enforcement comes knocking. This would, needless to say, be massively inefficient and cumbersome: The unique flexibility and resilience of the Internet comes precisely from the fact that it doesn’t depend on these sorts of centralized bottlenecks, which means my Chrome browser doesn’t suddenly become useless if Google’s servers go down, or become unreachable from my location for any reason. I don’t have to go through Mountain View, California, just to open a secure connection between my home in DC and my bank in Georgia. And, of course, it has the same problem as the previous approach: It creates a single point of catastrophic failure. An attacker who breaches the master key repository — or is able to successfully impersonate the repository — has hit the ultimate jackpot, and will invest whatever resources are necessary to do so.

Yet another option — perhaps the simplest — is to have the browser encrypt each session key with the same public key, either the developer’s or that of some government agency, and transmit it along with the communication. Then a law enforcement agency wanting to decrypt an intercepted HTTPS session goes to the developer or government entity whose public key was used to encrypt the session key, asks them to use their corresponding private key to unlock the session key, and uses that to decipher the actual communication. You can already guess the problem with this, right? That private key becomes a skeleton key that has to be kept secure against attackers, sacrificing the security advantages of forward secrecy. It has the additional problem of requiring a lot of additional anti-circumvention bells and whistles to prevent the user from trivially re-securing their session, since anyone who doesn’t want the new eavesdropping “feature” just has to install a plugin or some third-party application that catches the packet with the encrypted session key before it leaves the user’s machine.

There’s a further wrinkle: All complex software has vulnerabilities — like the Heartbleed bug in the widely-used OpenSSL library, which made headlines last year as exposing millions of users to the risk of having their secure communications compromised and spurred a frantic global patching effort. Modern software is never really finished: New vulnerabilities are routinely discovered, need to be patched, updates must be widely disseminated and installed, and then the cycle starts all over again. That’s hard enough as it is — an exercise in dancing madly on the lip of a volcano, as John Oliver memorably put it. Now there’s the added problem of ensuring that a new update to fix an unintentional vulnerability doesn’t simultaneously either break the intentional vulnerability introduced to provide law enforcement access, or interact unpredictably with any of the myriad approaches to guaranteeing government access in a way that creates a new vulnerability. People who don’t actually have to do this for a living are awfully confident this must be possible if the nerds only nerd hard enough. The actual nerds mostly seem to agree that it isn’t.

So far so awful. But now consider what this implies for the broader ecosystem — which doesn’t just consist of huge corporations like Apple or Google, but individual coders, small startups, and the open source communities that collaboratively produce software like Firefox. In principle, a lone computer science student, or a small team contributing to an open source project, can today write their own Web browser (or any other app implementing TLS) using open code libraries like OpenSSL, and release it online. We owe much of the spectacular innovation we’ve seen over the past few decades to the fact that software can be produced this way: You don’t even need a revenue model or a business plan or a corporate charter, just the knowledge to write code and the motivation to put it to use. Maybe the software makes you rich and launches the next Silicon Valley behemoth, or maybe the authors release it and forget about it, or move on and pass the torch to another generation of open source contributors.

If Feinstein-Burr becomes law, say goodbye to all that — at least when it comes to software that supports encryption of files or communications. Those existing open-code libraries don’t support government backdoors, so the developer will have to customize the code to support their chosen government access mechanism (with the attendant risk of introducing vulnerabilities in the process) and then be prepared to secure the master key material for the effective life of the software—that or run a server farm to act as a key repository and secure that. As a practical matter, the (lawful) production of secure software in the United States becomes the exclusive domain of corporate entities large and rich enough to support (and at least attempt to secure) some kind of key-escrow and law enforcement compliance infrastructure.

The probable upshot of this proposal, then, isn’t just that we all become less secure as big companies choose from a menu of terrible options that will enable them to comply with decryption orders — though it’s important to keep pointing that out, since legislators seem somehow convinced the experts are all lying about this. It’s that smaller developers and open source projects look at the legal compliance burdens associated with incorporating encryption in their software and decide it isn’t worth the trouble. Implementing crypto correctly is already hard enough; add the burden of designing and maintaining a Feinstein-Burr compliance strategy and a lot of smaller developers and open source projects are going to conclude it’s not worth the trouble.

In an environment of dire and growing cybersecurity threats, in other words, legislators seem determined to dissuade software developers from adopting better security practices. That would be a catastrophically bad idea, and urging developers to “nerd harder” doesn’t make it any less irresponsible.

Original URL:  

Original article

10nm versus 7nm

The silicon foundry business is heating up, as vendors continue to ramp their 16nm/14nm finFET processes. At the same time, they are racing each other to ship the next technologies on the roadmap—10nm and 7nm.

But the landscape is complicated, with each vendor taking a different strategy. Samsung, for one, plans to ship its 10nm finFET technology by year’s end. The company will focus on 10nm for now, although it also is working on 7nm in R&D.

In contrast, TSMC will move into 10nm production in early 2017, with 7nm slated to ship in 2018. TSMC sees 10nm as a shorter node and is emphasizing 7nm.

Meanwhile, Intel will move into 10nm production by mid-2017, with 7nm slated for 2018 or 2019, sources said. In addition, GlobalFoundries will also be in the mix at 7nm.

Based on those schedules and other factors, foundry customers that can afford to migrate to these advanced nodes face some tough decisions. Initially, customers may ask themselves a fundamental question: Does it makes more sense to migrate to either 10nm or 7nm. Or should they do both?

Over time, OEMs will embrace both 10nm and 7nm chips. But most foundry customers don’t have the resources to pursue both technologies, so they must weigh their options and make the right choices. Here are the main avenues for customers:

• Migrate from 16nm/14nm to 10nm.
• Skip 10nm and move directly to 7nm.
• Do both 10nm and 7nm.
• Stay at 16nm/14nm.

Another idea is to consider 2.5D stacked die or fan-out packaging.

The permutations are also complex for those at 28nm. If 28nm customers can afford it, they may move to 16nm/14nm and perhaps beyond. 22nm FD-SOI is another option. And, of course, if they can’t afford to migrate, they will stay put.

“This is going to be an interesting fork in the road for many customers,” said Kelvin Low, senior director of foundry marketing at Samsung Semiconductor. “There are some companies, particularly large companies, that will use every single node that the foundries introduce. The majority are kind of stepping back and evaluating whether that is the right recipe for success. We are starting to see a lot of small- to medium-size vendors skipping nodes today.”

For leading-edge foundry customers, it’s a complex decision to go with 10nm and/or 7nm. For example, both 10nm and 7nm are based on scaled down versions of today’s finFET transistors. But the definitions of these nodes are fuzzy, and not all 10nm and 7nm technologies are alike.

In theory, 7nm provide better performance than 10nm. But 10nm is expected to ship much sooner than 7nm.

Indeed, there are a multitude of tradeoffs. Cost, of course, is a big factor. “Everything is usually driven by economics,” said Matt Paggi, vice president of advanced technology at GlobalFoundries.

In fact, it costs $271 million to design a 7nm system-on-a-chip, which is about nine times the cost to design a 28nm device, according to Gartner. “Not that many people can afford to (design chips at 10nm and 7nm) unless they have a high-volume runner and can see a return-on-investment,” said Samuel Wang, an analyst with Gartner.

It’s also too expensive to make the wrong choice at 10nm and 7nm. Making the wrong bet can be disastrous.

To help foundry customers get ahead of the curve, Semiconductor Engineering has taken a look at the various tradeoffs at 10nm and 7nm.

The tradeoffs
Not long ago, it was a straightforward and inexpensive effort to migrate from one node to another. But the dynamics appeared to change at 16nm/14nm, when foundry vendors introduced finFET transistors for leading-edge designs.

FinFETs solve the short-channel effects that plague leading-edge planar devices. In finFETs, the control of the current is accomplished by implementing a gate on each of the three sides of a fin.

While finFETs keep the industry on the scaling path, the problem is that fewer and fewer foundry customers can afford the technology. In total, the average IC design cost for a 14nm chip is about $80 million, compared to $30 million for a 28nm planar device, according to Gartner.

For most, 28nm is suitable for many applications. “28nm will be a sweet spot for a long time, especially for cost and the ability to have the right power performance,” Samsung’s Low said.

Still, there is a sizable customer base that will migrate to 16nm/14nm and beyond. But in the early stages of 16nm/14nm, foundry vendors struggled with their yields. IC designers wrestled with the double patterning issues at that node.

Recently, though, the 16nm/14nm yields have improved. “Right now, there is less concern about the uncertainties of double patterning,” Low said. “All of that has gone away.”

Today, 16nm/14nm processes are more mature, which is fueling a broader adoption of the technology. “We are observing that 14nm could become a long node,” Low said. “There are not only the mobile and consumer guys that are enjoying the benefits of 14nm finFETs, but networking and server customers, as well.”

Amid the 16nm/14nm ramp, foundries are now pushing 10nm and 7nm. But needless to say, customers face some daunting challenges at those nodes. For one thing, there is some confusion between 10nm and 7nm.

“Not all 10nm technologies are the same,” said Mark Bohr, a senior fellow and director of process architecture and integration at Intel. “It’s now becoming clear that what other companies call a ‘10nm’ technology will not be as dense as Intel’s 10nm technology. We expect that what others call ‘7nm’ will be close to Intel’s 10nm technology for density.”

It wasn’t always like that. Traditionally, chipmakers scaled the key transistor specs by 0.7X at each node. This, in turn, roughly doubles the transistor density at each node.

Intel continues to follow this formula. At 16nm/14nm, though, others deviated from the equation from a density standpoint. For example, foundry vendors introduced finFETs at 16nm/14nm, but it incorporated a 20nm interconnect scheme.

Technically, the foundries didn’t introduce finFETs at a full node (14nm), but rather at a half node. TSMC, for one, calls it 16nm.

Still, foundries found a way to provide value to their customers at 16nm/14nm. “Foundries are less fixed on sticking to a 0.7X pitch shrink per node and more on providing their customers with some combination of power, performance, area and cost benefit at a half-node cadence,” said Mike Chudzik, senior director of strategic planning at Applied Materials.

There are other deviations from the roadmap. For example, the complexity of the technology is causing the cadence of the nodes to extend from their historical two-year patterns to roughly 2.5 years, analysts said.

With those issues in mind, customers must sort out the tradeoffs between the various foundry vendors at 10nm and 7nm. As before, they will look at the traditional metrics—performance, power, area scaling, schedule, and cost (PPASC).

The decision to select one process over another also depends on the product requirements. “Each process has different tradeoffs,” said Mark Liu, president and co-CEO of TSMC. “It also depends on the timing of the process.”

For many, it comes down to cost. For a 10nm chip, it takes $120 million for the design cost, plus 60% for the embedded software. In comparison, the total design cost is roughly $271 million for a 7nm chip, according to Gartner.

“It will take chip designers about 500 man-years to bring out a mid-range 7nm SoC to production,” Gartner’s Wang said. Therefore, a team of 50 engineers will need 10 years to complete the chip design to tape-out. In comparison, it could take 300 engineer-years to bring out a 10nm device, 200 for 14nm, and 100 for 28nm, according to Gartner.

Beside cost, there is other trouble on the horizon—the PPASC equation breaks down at 10nm and 7nm. “Because Moore’s Law has broken down and you no longer get gains in all areas at the same time by going to the next node, each foundry customer will have a different strategy, depending on which parameter is most important,” according to a source at one large customer, who asked not to be named.

Generally, customers could go down one of two paths for 10nm and 7nm, the source said. The first path is based on a combination of power and performance. The second is based on cost.

The power/performance path is for customers that will migrate directly from 16nm/14nm to 10nm. These customers require a new chip at each node in response to products with rapid design cycles, such as smartphones and PCs. This group includes suppliers of microprocessors and applications processors. Apple, Intel, Mediatek, Qualcomm and Samsung fall into this crowd.

Then, there is the cost-driven decision. This is for foundry customers who may have lower volume products. They may not recoup their investment at 10nm. So, it makes more sense for them to skip 10nm and move to 7nm.

Some FPGA vendors fall in this camp. IDMs and foundry customers that developed 10nm would also move to 7nm.

There is another way to look at the various scenarios. “This is going to be exactly the same thing as 20nm. 20nm was a transition node. It only served a few customers,” Gartner’s Wang said.

“10nm will be the same thing as 20nm. Customers who used 20nm before will most likely use 10nm,” Wang said. “Customers that skipped 20nm, and went directly to 16nm, will most likely do the same thing. They will skip 10nm and go to 7nm.”

All told, IDMs and foundry customers will simultaneously ship chips at several different nodes. “Chipmakers targeting different applications and serving different markets will have varying strategies in adopting these leading-edge nodes,” said Yang Pan, chief technology officer for the Global Products Group at Lam Research.

“10nm is expected to start ramping within the next 12 months, and 7nm development is progressing at full speed,” Pan said. “With rising development costs and the increasing cost to migrate designs from one node to the next, we anticipate more overlap between nodes as the industry tries to maximize returns from investments made at each node. Chipmakers targeting different applications and serving different markets will have varying strategies in adopting these leading-edge nodes.”

What is 10nm?
Still, the decisions aren’t quite that simple. In fact, there are two schools of thought at 10nm and 7nm. One camp says that 10nm will be a robust node, while others say 7nm will become the more dominate node.

Samsung is in the first camp. “We see 10nm as a very healthy node. It’s probably going to be widely adopted. It has the right PPASC,” Samsung’s Low said. “We don’t see a need to rush into 7nm. 7nm has to be properly defined, taking cost into account.”

According to officials from Samsung, 193nm immersion lithography and multi-patterning will extend to at least 10nm. With those patterning technologies in place, foundry customers and IDMs can develop cost-effective IC designs at 10nm.

But at 7nm, optical and multi-patterning are simply too complex and expensive, at least according to Samsung. So to make 7nm cost effective, it makes more sense to wait for EUV. In theory, EUV can simplify the patterning process.

“EUV has been delayed for a long time. During that time, 193nm immersion has been the workhorse for the semiconductor industry,” added Seong-Sue Kim, a technical staff member within the Semiconductor R&D Center at Samsung. “But in the case of 7nm, the situation is different. Of course, 193nm immersion has (advanced) technologically, but the problem is cost. The situation is we need EUV.”

TSMC, however, has a different strategy. “(TSMC) is also aggressive in terms of expanding into 10nm,” Gartner’s Wang said, “but 10nm is not considered as a main technology by TSMC. They actually are pushing hard on 7nm.”

Foundry vendors, meanwhile, are keeping their 10nm specs close to the vest. In general, a 10nm finFET would include the usual features, such as copper interconnects and high-k/metal-gate. It would make use of 193nm immersion and multiple patterning.

As it stands today, 10nm will ship by year’s end, roughly a year ahead of 7nm. 10nm has a 22% speed improvement over 16nm/14nm.

There are some disadvantages with 10nm, though. “I believe a few key customers looked at the 10nm process offering from TSMC and decided it will be better to wait for the 7nm solution,” said Joanne Itow, an analyst with Semico Research. “10nm will not get enough improvement compared to all the time and money required.”

There are other tradeoffs. TSMC is moving from a 2D layout scheme at 16nm to 1D technology at 10nm. 1D layouts are easier to make in the fab, but they involve more restrictive design rules.

Others may follow the 2D layout path at 10nm. This technology is harder to make in the fab, but it enables a more flexible design environment.

“Up through 16nm, all the foundry approaches to multi-patterning have been very similar,” said David Abercrombie, advanced physical verification methodology program manager at Mentor Graphics. “Beginning with 10nm, those approaches have diverged.

Correspondingly, the specific types of checks, coloring requirements, error visualization and design considerations are different between them.”

What is 7nm?
Like 10nm, 7nm has some pluses and minuses. Compared to 16nm/14nm, 7nm provides a 35% speed improvement, 65% less power, and a 3.3X density improvement, according to Gartner.

Based on PPASC metrics and the cost-per-transistor curve, 7nm looks like a better option, at least according to some. “In general, we are seeing that this financial equation is pretty tight for most customers at 10nm,” GlobalFoundries’ Paggi said. “7nm, for most customers in most of the markets, appears to be a more favorable financial equation.”

There is room for 10nm, at least for some applications. “But for the largest part of the market, we see that the economics of 7nm is compelling enough,” Paggi said. “And at 10nm, it’s marginal.

“(7nm) brings a lot larger economic benefits,” he said. “In most cases, that outweighs the design cost you would have to spend. There are power advantages as well as a longer battery life.”

Still, 7nm presents some major challenges. Like 10nm, 7nm is a scaled version of a finFET. Originally, chipmakers hoped to insert extreme ultraviolet (EUV) lithography at 7nm. But it’s unlikely EUV will intersect the early part of 7nm, meaning that chipmakers must use complex multiple patterning schemes at that node.

What will this all mean for IC designers at both 10nm and 7nm? “These nodes first introduce constraints and checking for triple, quad litho-etch based and self-aligned double-patterning based processes,” Mentor’s Abercrombie said. “So designers will need to learn these new concepts and checks associated with them. In addition, there is a general trend to more constrained layouts. There is a general move towards track and grid based layout forms. Expect this trend to increase moving forward.”

Related Stories
Pain Points At 7nm
7nm Lithography Choices
The Bumpy Road To 10nm FinFETs

Original URL:  

Original article

Go best practices, six years in

(This article was originally a talk at QCon London 2016. Slides and video coming soon.)

In 2014, I gave a talk at the inaugural GopherCon titled Best Practices in Production Environments.
We were early adopters at SoundCloud, and by that point had been writing, running, and maintaining Go in production in one form or another for nearly 2 years.
We had learned a few things, and I tried to distill and convey some of those lessons.

Since then, I’ve continued working in Go full-time,
later on the activities and infrastructure teams at SoundCloud,
and now at Weaveworks,
on Weave Scope
and Weave Mesh.
I’ve also been working hard on Go kit,
an open-source toolkit for microservices.
And all the while, I’ve been active in the Go community,
meeting lots of developers at meetups and conferences throughout Europe and the US,
and collecting their stories—both successes and failures.

With the 6th anniversary of Go’s release in November of 2015,
I thought back to that first talk.
Which of those best practices have stood the test of time?
Which have become outmoded or counterproductive?
Are there any new practices that have emerged?
In March, I had the opportunity to give a talk at QCon London
where I reviewed the best practices from 2014 and took a look at how Go has evolved in 2016.
Here’s the meat of that talk.

I’ve highlighted the key takeaways as linkable Top Tips.

Top Tip
Use Top Tips to level up your Go game.

And a quick table of contents…

  1. Development environment
  2. Repository structure
  3. Formatting and style
  4. Configuration
  5. Program design
  6. Logging and instrumentation
  7. Testing
  8. Dependency management
  9. Build and deploy
  10. Conclusion

Go has development environment conventions centered around the GOPATH.
In 2014 I advocated strongly for a single global GOPATH.
My positioned has softened a bit.
I still think that’s the best idea, all else equal, but depending on your project or team, other things may make sense, too.

If you or your organization produces primarily binaries, you might find some advantages with a per-project GOPATH.
There’s a new tool, gb, from Dave Cheney and contributors, which replaces the standard go tooling for this use-case.
A lot of people are reporting a lot of success with it.

Some Go developers use a two-entry GOPATH, e.g. $HOME/go/external:$HOME/go/internal.
The go tool has always known how to deal with this: go get will fetch into the first path,
so it can be useful if you need strict separation of third-party vs. internal code.

One thing I’ve noticed some developers forget to do:
put GOPATH/bin into your PATH.
This allows you to easily run binaries you get via go get,
and makes the (preferred) go install mechanism of building code easier to work with.
No reason not to do it.

Top Tip
Put $GOPATH/bin in your $PATH, so installed binaries are easily accessible.

Regarding editors and IDEs, there’s been a lot of steady improvement.
If you’re a vim warrior, life has never been better:
thanks to the tireless and extremely capable efforts of Fatih Arslan,
the vim-go plugin is in an absolutely exceptional state,
I’m not as familiar with emacs, but Dominik Honnef’s
go-mode.el is still the big kahuna there.

Moving up the stack, lots of folks are still using and having success with
Sublime Text + GoSublime.
And it’s hard to beat the speed.
But more attention seems to be paid lately to the Electron-powered editors.
Atom + go-plus has many fans,
especially those developers that have to frequently switch languages to JavaScript.
The dark horse has been Visual Studio Code
+ vscode-go, which,
while slower than Sublime Text, is noticably faster than Atom,
and has excellent default support for important-to-me features,
like click-to-definition.
I’ve been using it daily for about half a year now,
after being introduced to it by Thomas Adam.
Lots of fun.

In terms of full IDEs, the purpose-built LiteIDE
has been receiving regular updates and certainly has its share of fans.
And the IntelliJ Go plugin
has been consistently improving as well.

We’ve had a lot of time for projects to mature, and some clear patterns have emerged.
How you structure your repo depends on what your project is.
First, if your project is private or internal to your company, go nuts:
have it represent its own GOPATH,
use a custom build tool,
whatever makes you happy and productive.

However, if your project is public (i.e. open-source), the rules get a little stricter.
You should play nice with go get,
because that’s still how most Go developers will want to consume your work.

Ideal structure depends on the type of your artifacts.
If your repo is exclusively a binary or a library,
then make sure consumers can go get or import it by its base path.
That is, put the package main or primary importable resource in,
and use subdirectories for helper packages.

If your repo is a combination of binaries and libraries,
then you should determine which is the primary artifact,
and put that in the root of the repo.
For example, if your repo is primarily a binary, but can also be used as a library,
then you’ll want to structure it like this:
    main.go      // package main
    main_test.go // package main
        foo.go      // package foo
        foo_test.go // package foo

One useful thing is to name the package in the lib/ subdirectory for the library rather than the directory,
in this case making it package foo rather than package lib.
This is an exception to an otherwise pretty strict Go idiom,
but in practice, it’s very friendly for consumers.
One great repo that’s laid out this way is tsenart/vegeta,
an HTTP load testing tool.

Top Tip
If your repo foo is primarily a binary, put your library code in a lib/ subdir, and call it package foo.

If your repo is primarily a library, but it also includes one or more binaries,
then you’ll want to structure it like this:
    foo.go      // package foo
    foo_test.go // package foo
            main.go      // package main
            main_test.go // package main

You’ll invert the structure, putting the library code at the root,
and making a cmd/foo/ subdirectory to hold the binary code.
The cmd/ intermediary is nice for two reasons:
the Go tooling automatically names binaries after the directory of their package main,
so it allows us to give them the best possible names without potentially conflicting
with other packages in the repo;
and it makes it unambiguous to your consumers what they’re getting
when they go get a path with /cmd/ in the middle.
The build tool gb is laid out this way.

Top Tip
If your repo is primarily a library, put your binaries in separate subdirectories under cmd/.

The idea here is to optimize for consumers:
make it easy to consume the most common form of your project.
This abstract idea, a focus on consumers, is I think part of the Go ethos.

Things have stayed largely the same here.
This is one area that Go has gotten quite right,
and I really appreciate the consensus in the community and stability in the language.

The Code Review Comments are great,
and should be the minimum set of critera you enforce during code review.
And when there are disputes or inconsistencies in names,
Andrew Gerrand’s idiomatic naming conventions
are a great set of guidelines.

And in terms of tooling, things have only gotten better.
You should configure your editor to invoke gofmt on save.
(At this point, I hope that’s not in any way controversial.)
To the best of my knowledge,
the go vet tool produces no false positives,
so it’s a good idea to make it part of your precommit hook.
And check out the excellent gometalinter for linting concerns.
This can produce false positives, so it’s not a bad idea to
encode your own conventions somehow.

Configuration is the surface area between the runtime environment and the process.
It should be explicit and well-documented.
I still use and recommend package flag, but I admit at this point I wish it were less esoteric.
I wish it had standard, getopts-style long- and short-form argument syntax,
and I wish its usage text were much more compact.

12-factor apps encourage you to use environment vars for configuration,
and I think that’s fine, provided each var is also defined as a flag.
Explicitness is important: changing the runtime behavior of an application
should happen in ways that are discoverable and documented.

I said it in 2014 but I think it’s important enough to say again:
define and parse your flags in func main.
Only func main has the right to decide the flags that will be available to the user.
If your library code wants to parameterize its behavior,
those parameters should be part of type constructors.
Moving configuration to package globals has the illusion of convenience, but it’s a false economy:
doing so breaks code modularity,
makes it more difficult for developers or future maintainers to understand dependency relationships,
and makes writing independent, parallelizable tests much more difficult.

Top Tip
Only func main has the right to decide which flags are available to the user.

I think there’s a great opportunity for a well-scoped flags package to emerge from the community,
combining all of these characteristics.
Maybe it already exists; if so, please let me know.
I’d certainly use it.

In the talk, I used configuration as a jumping-off point, to discuss a few other issues of program design.
(I didn’t cover this in the 2014 version.)
To start, let’s take a look at constructors.
If we are properly parameterizing all of our dependencies, our constructors can get quite large.

foo, err := newFoo(
    100 * time.Millisecond,
if err != nil {
defer foo.close()

Sometimes this kind of construction is best expressed with a config object:
a struct parameter to a constructor that takes optional parameters to the constructed object.
Let’s assume fooKey is a required parameter,
and everything else either has a sensible default or is optional.
Often, I see projects construct config objects in a sort of piecemeal way.

// Don't do this.
cfg := fooConfig{}
cfg.Bar = bar
cfg.Period = 100 * time.Millisecond
cfg.Output = nil

foo, err := newFoo(*fooKey, cfg)
if err != nil {
defer foo.close()

But it’s considerably nicer to leverage so-called struct initialization syntax
to construct the object all at once, in a single statement.

// This is better.
cfg := fooConfig{
    Bar:    bar,
    Period: 100 * time.Millisecond,
    Output: nil,

foo, err := newFoo(*fooKey, cfg)
if err != nil {
defer foo.close()

No statements go by where the object is in an intermediate, invalid state.
And all of the fields are nicely delimited and indented, mirroring the fooConfig definition.

Notice we construct and then immediately use the cfg object.
In this case we can save another degree of intermediate state, and another line of code,
by inlining the struct declaration into the newFoo constructor directly.

// This is even better.
foo, err := newFoo(*fooKey, fooConfig{
    Bar:    bar,
    Period: 100 * time.Millisecond,
    Output: nil,
if err != nil {
defer foo.close()


Top Tip
Use struct literal initialization to avoid invalid intermediate state.
Inline struct declarations where possible.

Let’s turn to the subject of sensible defaults.
Observe that the Output parameter is something that can take a nil value.
For the sake of argument, assume it’s an io.Writer.
If we don’t do anything special, when we want to use it in our foo object,
we’ll have to first perform a nil check.

func (f *foo) process() {
    if f.Output != nil {
        fmt.Fprintf(f.Output, "startn")
    // ...

That’s not great.
It’s much safer, and nicer, to be able to use output without having to check it for existence.

func (f *foo) process() {
     fmt.Fprintf(f.Output, "startn")
     // ...

So we should provide a usable default here.
With interface types, one good way is to pass something that provides a no-op implementation of the interface.
And it turns out that the stdlib ioutil package comes with a no-op io.Writer, called ioutil.Discard.

Top Tip
Avoid nil checks via default no-op implementations.

We could pass that into the fooConfig object, but that’s still fragile.
If the caller forgets to do it at the callsite, we’ll still end up with a nil parameter.
So, instead, we can create a sort of safety within the constructor.

func newFoo(..., cfg fooConfig) *foo {
    if cfg.Output == nil {
        cfg.Output = ioutil.Discard
    // ...

This is just an application of the Go idiom make the zero value useful.
We allow the zero value of the parameter (nil) to yield good default behavior (no-op).

Top Tip
Make the zero value useful, especially in config objects.

Let’s revisit the constructor.
The parameters fooKey, bar, period, output are all dependencies.
The foo object depends on each of them in order to start and run successfully.
If there’s a single lesson I’ve learned from writing Go code in the wild
and observing large Go projects on a daily basis for the past six years, it is this:
make dependencies explicit.

Top Tip
Make dependencies explicit!

An incredible amount of maintenance burden, confusion, bugs, and unpaid technical debt
can, I believe, be traced back to ambiguous or implicit dependencies.
Consider this method on the type foo.

func (f *foo) process() {
    fmt.Fprintf(f.Output, "startn")
    result := f.Bar.compute()
    log.Printf("bar: %v", result) // Whoops!
    // ...

fmt.Printf is self-contained and doesn’t affect or depend on global state;
in functional terms, it has something like referential transparency.
So it is not a dependency.
Obviously, f.Bar is a dependency.
And, interestingly, log.Printf acts on a package-global logger object,
it’s just obscured behind the free function Printf.
So it, too, is a dependency.

What do we do with dependencies?
We make them explicit.
Because the process method prints to a log as part of its work,
either the method or the foo object itself needs to take a logger object as a dependency.
For example, log.Printf should become f.Logger.Printf.

func (f *foo) process() {
    fmt.Fprintf(f.Output, "startn")
    result := f.Bar.compute()
    f.Logger.Printf("bar: %v", result) // Better.
    // ...

We’re conditioned to think of certain classes of work, like writing to a log, as incidental.
So we’re happy to leverage helpers, like package-global loggers, to reduce the apparent burden.
But logging, like instrumentation, is often crucial to the operation of a service.
And hiding dependencies in the global scope can and does come back to bite us,
whether it’s something as seemingly benign as a logger,
or perhaps another, more important, domain-specific component that we haven’t bothered to parameterize.
Save yourself the future pain by being strict: make all your dependencies explicit.

Top Tip
Loggers are dependencies, just like references to other components,
database handles, commandline flags, etc.

Of course, we should also be sure to take a sensible default for our logger.

func newFoo(..., cfg fooConfig) *foo {
    // ...
    if cfg.Logger == nil {
        cfg.Logger = log.NewLogger(ioutil.Discard, ...)
    // ...

To speak about the problem generally for a moment:
I’ve had a lot more production experience with logging,
which has mostly just increased my respect for the problem.
Logging is expensive, more expensive than you think,
and can quickly become the bottleneck of your system.
I wrote more extensively on the subject in a separate blog post,
but to re-cap:

  • Log only actionable information, which will be read by a human or a machine
  • Avoid fine-grained log levels — info and debug are probably enough
  • Use structured logging — I’m biased, but I recommend go-kit/log
  • Loggers are dependencies!

Where logging is expensive, instrumentation is cheap.
You should be instrumenting every significant component of your codebase.
If it’s a resource, like a queue, instrument it according to
Brendan Gregg’s USE method:
utilization, saturation, and error count (rate).
If it’s something like an endpoint, instrument it according to
Tom Wilkie’s RED method:
request count (rate), error count (rate), and duration.

If you have any choice in the matter,
Prometheus is probably the instrumentation system you should be using.
And, of course, metrics are dependencies, too!

Let’s use loggers and metrics to pivot and address global state more directly.
Here are some facts about Go:

  • log.Print uses a fixed, global log.Logger
  • http.Get uses a fixed, global http.Client
  • http.Server, by default, uses a fixed, global log.Logger
  • database/sql uses a fixed, global driver registry
  • func init exists only to have side effects on package-global state

These facts are convenient in the small, but awkward in the large.
That is, how can we test the log output of components that use the fixed global logger?
We must redirect its output, but then how can we test in parallel?
Just don’t?
That seems unsatisfactory.
Or, if we have two independent components both making HTTP requests with different requirements,
how do we manage that?
With the default global http.Client, it’s quite difficult.
Consider this example.

func foo() {
    resp, err := http.Get("")
    // ...

http.Get calls on a global in package http.
It has an implicit global dependency.
Which we can eliminate pretty easily.

func foo(client *http.Client) {
    resp, err := client.Get("")
    // ...

Just pass an http.Client as a parameter.
But that is a concrete type, which means if we want to test this function
we also need to provide a concrete http.Client,
which likely forces us to do actual HTTP communication.
Not great.
We can do one better, by passing an interface which can Do (execute) HTTP requests.

type Doer interface {
    Do(*http.Request) (*http.Response, error)

func foo(d Doer) {
    req, _ := http.NewRequest("GET", "", nil)
    resp, err := d.Do(req)
    // ...

http.Client satisfies our Doer interface automatically,
but now we have the freedom to pass a mock Doer implementation in our test.
And that’s great: a unit test for func foo is meant to test only the behavior of foo,
it can safely assume that the http.Client is going to work as advertised.

Speaking of testing…

In 2014, I reflected on our experience with various testing frameworks and helper libraries,
and concluded that we never found a great deal of utility in any of them,
recommending the stdlib’s approach of plain package testing with table-based tests.
Broadly, I still think this is the best advice.
The important thing to remember about testing in Go is that it is just programming.
It is not sufficiently different from other programming that it warrants its own metalanguage.
And so package testing continues to be well-suited to the task.

TDD/BDD packages bring new, unfamiliar DSLs and control structures,
increasing the cognitive burden on you and your future maintainers.
I haven’t personally seen a codebase where that cost has paid off in benefits.
Like global state, I believe these packages represent a false economy,
and more often than not are the product of cargo-culting behaviors from other languages and ecosystems.
When in Go, do as Gophers do: we already have a language for writing simple,
expressive tests—it’s called Go, and you probably know it pretty well.

With that said, I do recognize my own context and biases.
Like with my opinions on the GOPATH, I’ve softened a bit,
and defer to those teams and organizations for whom a testing DSL or framework may make sense.
If you know you want to use a package, go for it.
Just be sure you’re doing it for well-defined reasons.

Another incredibly interesting topic has been designing for testing.
Mitchell Hashimoto recently gave a great talk on the subject here in Berlin
which I think should be be required viewing.

In general, the thing that seems to work the best is to write Go in a generally functional style,
where dependencies are explicitly enumerated, and provided as small, tightly-scoped interfaces whenever possible.
Beyond being good software engineering discipline in itself,
it feels like it automatically optimizes your code for easy testing.

Top Tip
Use many small interfaces to model dependencies.

As in the http.Client example just above, remember that unit tests should be written to
test the thing being tested, and nothing more.
If you’re testing a process function,
there’s no reason to also test the HTTP transport the request came in on,
or the path on disk the results get written to.
Provide inputs and outputs as fake implementations of interface parameters,
and focus on the business logic of the method or component exclusively.

Top Tip
Tests only need to test the thing being tested.

Ever the hot topic.
In 2014, things were nascent, and about the only concrete advice I could give was to vendor.
That advice still holds today: vendoring is still the solution to dependency management for binaries.
In particular, the GO15VENDOREXPERIMENT and its concomittant vendor/ subdirectory
have become default in Go 1.6.
So you’ll be using that layout.
And, thankfully, the tools have gotten a lot better.
Some I can recommend:

  • FiloSottile/gvt takes a minimal approach,
    basically just extracting the vendor subcommand from the gb tool
    so it can be used standalone.
  • Masterminds/glide takes a maximal approach,
    attempting to recreate the feel and finish of a fully-featured dependency management tool
    using vendoring under the hood.
  • kardianos/govendor sits in the middle,
    providing probably the richest interface to vendoring-specific nouns and verbs,
    and is driving the conversation on the manifest file.
  • constabulary/gb abandons the go tooling altogether
    in favor of a different repository layout and build mechanism.
    Great if you produce binaries and can mandate the build environment, e.g. in a corporate setting.
Top Tip
Use a top tool to vendor dependencies for your binary.

A big caveat for libraries.
In Go, dependency management is a concern of the binary author.
Libraries with vendored dependencies are very difficult to use;
so difficult that it is probably better said that they are impossible to use.
There are many corner cases and edge conditions that have played out in the months since
vendoring was officially introduced in 1.5.
(You can dig in to one of these
forum posts
if you’re particularly interested in the details.)
Without getting too deep in the weeds, the lesson is clear:
libraries should never vendor dependencies.

Top Tip
Libraries should never vendor their dependencies.

You can carve out an exception for yourself if your library has hermetically sealed its dependencies,
so that none of them escape to the exported (public) API layer.
No dependent types referenced in any exported functions, method signatures, structures—anything.

If you have the common task of maintaining an open-source repository
that contains both binaries and libraries,
unfortunately, you are stuck between a rock and a hard place.
You want to vendor your deps for your binaries,
but you shouldn’t vendor them for your libraries,
and the GO15VENDOREXPERIMENT doesn’t admit this level of granularity,
from what appears to me to be regrettable oversight.

Bluntly, I don’t have an answer to this.
The etcd folks have hacked together a solution using symlinks
which I cannot in good faith recommend,
as symlinks are not well-supported by the go toolchain and break entirely on Windows.
That this works at all is more a happy accident than any consequence of design.
I and others have raised all of these concerns to the core team,
and I hope something will happen in the near term.

Regarding building, one important lesson learned, with a hat tip to Dave Cheney:
prefer go install to go build.
The install verb caches build artifacts from dependencies in $GOPATH/pkg,
making builds faster.
It also puts binaries in $GOPATH/bin,
making them easier to find and invoke.

Top Tip
Prefer go install to go build.

If you produce a binary, don’t be afraid to try out new build tools
like gb, which may significantly reduce your cognitive burden.
Conversely, remember that since Go 1.5 cross-compilation is built-in;
just set the appropriate GOOS and GOARCH environment variables, and
invoke the appropriate go command.
So there’s no need for extra tools here anymore.

Regarding deployment, we Gophers have it pretty easy
compared to languages like Ruby or Python,
or even the JVM.
One note: if you deploy in containers,
follow the advice of Kelsey Hightower and do it FROM scratch.
Go gives us this incredible opportunity; it’s a shame not to use it.

As more general advice, think carefully before choosing a platform
or orchestration system—if you even choose one at all.
Likewise for jumping onto the microservices bandwagon.
An elegant monolith, deployed as an AMI to an autoscaling EC2 group,
is a very productive setup for small teams.
Resist, or at least carefully consider, the hype.

The Top Tips:

  1. Put $GOPATH/bin in your $PATH, so installed binaries are easily accessible.  link
  2. If your repo foo is primarily a binary, put your library code in a lib/ subdir, and call it package foo.  link
  3. If your repo is primarily a library, put your binaries in separate subdirectories under cmd/.  link
  4. Defer to Andrew Gerrand’s naming conventions.  link
  5. Only func main has the right to decide which flags are available to the user.  link
  6. Use struct literal initialization to avoid invalid intermediate state.  link
  7. Avoid nil checks via default no-op implementations.  link
  8. Make the zero value useful, especially in config objects.  link
  9. Make dependencies explicit!  link
  10. Loggers are dependencies, just like references to other components, database handles, commandline flags, etc.  link
  11. Use many small interfaces to model dependencies.  link
  12. Tests only need to test the thing being tested.  link
  13. Use a top tool to vendor dependencies for your binary.  link
  14. Libraries should never vendor their dependencies.  link
  15. Prefer go install to go build.  link

Go has always been a conservative language,
and its maturity has brought relatively few surprises and effectively no major changes.
Consequently, and predictably, the community also hasn’t dramatically shifted its stances
on what’s considered best practice.
Instead, we’ve seen a reification of tropes and proverbs
that were reasonably well-known in the early years,
and a gradual movement “up the stack” as design patterns, libraries, and program structures
are explored and transformed into idiomatic Go.

Here’s to another 6 years of fun and productive Go programming.

Original URL:  

Original article

Microsoft’s IE loses top browser spot to Google’s Chrome

Microsoft’s Internet Explorer (IE) last month lost the No. 1 spot to Google’s Chrome, marking a major milestone not only in IE’s 21-year lifespan, but a dramatic changing of the desktop browser guard.

According to U.S. analytics vendor Net Applications, IE and Edge — which the firm tossed into a single bucket labeled “IE” — fell 2 percentage points in April, the fifth straight month of a loss greater than a point, and the 16th of any size — to end at 41.4% of the total global browser user share.

Meanwhile, Chrome climbed 2.6 percentage points to take a narrow lead with 41.7%.

Previously, Computerworld had forecast — using long-term trends portrayed by Net Applications’ data — that Chrome would wrestle the No. 1 position from IE by the end of May.

To read this article in full or to leave a comment, please click here

Original URL:  

Original article

Introducing the Infinit file system

TL;DR: We launched a new product that allows creating reliable, secure and scalable storage infrastructure:

Over the last few months, we’ve been secretly working on a different, complementary product to what we built in the file transfer world. But back to the beginning.

About five years ago, Infinit was born out of Julien Quintard’s PhD thesis: Towards a worldwide storage infrastructure. Freshly out of his PhD, Julien co-founded the company that you know as Infinit. The team quickly took some elements of Julien’s PhD and built the file transfer application that many of you use daily.

The company’s vision however, had always been about revolutionizing the file storage industry. We therefore got back to work on our world conquest file storage master plan.

Infinit File Storage Platform


Our approach to file storage has been to consider storage resources and providers as third-party components, potentially not under the administrator’s control. As such, Infinit considers storage local resources and cloud providers to be untrustworthy, making no assumption regarding their nature (hardware agnosticism), reliability or behavior.

This approach coupled to a peer-to-peer i.e decentralized model allows for the creation of fault-tolerant, scalable yet affordable and easily deployable storage infrastructure composed of the aggregation of various storage capacities. And because we believe that no one other than yourself should be in control of your own data, whatever some people tend to say, we decided that Infinit would be designed with privacy and fault tolerance at its heart.

While most solutions (Dropbox, GlusterFS, OwnCloud etc.) store your files unprotected in the cloud or on a specific server, we took a diametrically opposed direction by relying heavily on encryption. Whenever a file is stored in Infinit, it is cut into chunks, every chunk is encrypted with a unique key (AES-256) and stored, providing both encryption at rest and in transit. Note that none of the data blocks composing your files ever end up on one of our servers. What you do with your data is your business!

Access control is another mechanism which differs from the common client/server model in which a specific server is requested to know if a client can access a file. In Infinit, there are no such super-privileged servers. Access control is handled through cryptography. Every user has a pair of RSA key (2048 bit or more). Whenever a file is modified, the modification is signed in order to ensure non-repudiation while a metadata-embedded access control mechanism is used to check the validity of future modifications against a list of authorized users.

The absence of such metadata and control servers allows for better scalability and resilience, removing any bottleneck and single point of failure.


With that in mind, rather than coldly choosing between the consumer or the enterprise world, we focused on building a series of tools for developers to set up and administer their own storage infrastructure. As a result, Infinit can act as a general-purpose platform for other applications to be built upon, for end-users to put their resources in common but also for enterprises to create complex storage infrastructure while remaining in control of their data.

If software is eating the world, developers are certainly shaping it! We therefore decided to embrace this philosophy, following other great companies and products before us.

The same thinking was applied when we decided whether Infinit would be a private storage solution or a worldwide network in the vein of the now-discontinued Wuala. In order to fit enterprise needs, Infinit had to provide a way to create controlled storage infrastructure. Nothing however prevents someone from using Infinit to create a global file system to which users all around the world contribute their unused storage capacity to create a massively distributed and secure Dropbox-like service, at a fraction of the cost.


For now, our platform requires a technical background i.e. being able to use command-line tools to manipulate Infinit file systems. Once an Infinit storage network created, the owner can invite his/her friends or colleagues. Because these people are likely to be less tech-savvy, we also offer a simplified desktop client for Mac, Windows and Linux that people can use to easily access their files like they do with Dropbox. And we decided to originally name it: Infinit Drive.

Open Source

You may think that it all sounds nice but that without the source code being open, there is no way you can be sure the system is actually secure. And you would be right! Our goal is to open source all the file system code. Everyone in our team has been using open-source software since we started programming. We believe in the power of open source and feel that it is only natural to give back to the community and be as transparent as possible regarding our technology. We hope that by open-sourcing our code, more developers will join us and contribute to improve the core technology and build awesome tools on top of it.

We started with our build system that we open-sourced a few days ago. We will continue open-sourcing more libraries in the coming weeks. We want every open-sourced part to be as modular and clean as possible for the community to use it.

We are proud to publicly introduce our new Infinit File Storage Platform. We are excited to see how people will use it! Have a look at the 5-minute tutorial, browse the FAQ or contact us.

Original URL:  

Original article

Proudly powered by WordPress | Theme: Baskerville 2 by Anders Noren.

Up ↑

%d bloggers like this: