GitHub’s Metal Cloud

At GitHub we place an emphasis on stability, availability, and performance. A large component of ensuring we excel in these areas is deploying services on bare-metal hardware. This allows us to tailor hardware configurations to our specific needs, guarantee a certain performance profile, and own the availability of our systems from end to end.

Of course, operating our own data centers and managing the hardware that’s deployed there introduces its own set of complications. We’re now tasked with tracking, managing, and provisioning physical pieces of hardware — work that is completely eliminated in a cloud computing environment. We also need to retain the benefits that we’ve all come to know and love in cloud environments: on-demand compute resources that are a single API call away.

Enter gPanel, our physical infrastructure management application.

The Application

gPanel Chassis

gPanel is a Ruby on Rails application that we started developing over three years ago as we were transitioning from a managed environment to our own data center space. It was identified early on that we’d need the ability to track physical components of our new space; cabinets, PDUs, chassis, switches, and loose pieces of hardware. With this in mind, we set out building the application.

As we started transitioning hosts and services to our own data center, we quickly realized we’d also need an efficient process for installing and configuring operating systems on this new hardware. This process should be completely automated, allowing us to make it accessible to the entire company. Without this, specific knowledge about our new environment would be required to spin up new hosts, which leaves the very large task of a complete data center migration exclusively in the hands of our small Operations team.

Since we’d already elected to have gPanel act as the source of truth for our data center, we determined it should be responsible for server provisioning as well.

The Pipeline

The system we ended up with is overall pretty straight-forward and simple — goals for any of our new systems or software. We utilize a few key pieces to drive the entire process.

Our hardware vendor configures machines to PXE boot from the network before they arrive at our data center. Machines are racked, connected to our network, and powered on. From there, our DHCP/PXE server tells the machines to chainload iPXE and then contact gPanel for further instructions. gPanel can identify the server (or determine that it’s brand new) with the serial number that’s passed as a parameter in the iPXE request.

gPanel defines a number of states that chassis are in. This state is passed to our Ubuntu PXE image via kernel parameters so it can determine which action to take. These actions are driven by a simple set of bash scripts that we include in our Ubuntu image.

The initial state is unknown where we simply collect data about the machine and record it in gPanel. This is accomplished using Facter for gathering system information, exporting it as JSON, and then POSTing it to gPanel’s API. gPanel has a number of jobs that process this JSON and create the appropriate records. We try to model as much as possible in the application; CPUs, DIMMs, RAID cards, drives, NICs, and more are all separate records in the database. This allows us to track parts as they’re replaced, moved to a different machine, or removed entirely.

Once we’ve gathered all the information we need about the machine, we enter configuring, where we assign a static IP address to the IPMI interface and tweak our BIOS settings. From there we move to firmware_upgrade where we update FCB, BMC, BIOS, RAID, and any other firmware we’d like to manage on the system.

At this point we consider the initial hardware configuration complete and will begin the burn-in process. Our burn-in process consists of two states in gPanel; breakin and memtesting. breakin uses a suite from Advanced Clustering to exercise the hardware and detect any problems. We’ve added a script that POSTs updates to gPanel throughout this process so it can determine whether we have failures or not. If a failure is detected, the chassis is moved to our failed state where it sits until we have a chance to review the logs and replace the bad component. If the chassis passes breakin, we’ll move on to memtesting.

In memtesting we boot a custom MemTest86 image and monitor it while it completes a full pass. Our custom version of MemTest86 changes the color of the failure message to red which allows us to detect trouble. We’ve hacked together a Ruby script that retrieves a console screenshot via IPMI and checks the color in the image to determine if we’ve hit a failure or not. Again, if a failure is detected, we’ll transition the chassis to failed, otherwise it moves on to ready.

The ready state is where our available pool of machines will sit until someone comes along and brings it into production.

The Workflow

Once machines have completed the burn-in process and deemed ready for production service, a user can instruct gPanel to install an operating system. Like the majority of our tooling, this is driven via Hubot, our programmable chat bot.


[gPanel] booting Dell Inc. PowerEdge R730 CKEJQIA (CKEJQIA) into ready…

First, the user will need to determine which chassis they’d like to perform the installation on.


/gpanel chassis list

Once the chassis is selected, you can initiate the installation.


/gpanel host install chassis=FUJGECQ


[gPanel] Installing OS on FUJGECQ as

[gPanel] Deploying DNS via Heaven…

hubot is deploying dns/master (deadbeef) to production.

hubot’s production deployment of dns/master (deadbeef) is done! (6s)

[gPanel] booting Dell Inc. PowerEdge C5220 FUJGECQ (github-worker42-cp1-prd) into installing…

If the user needs a different RAID configuration, or to have the host brought up on a different Puppet branch, they can specify those with the install command as well.


/gpanel host install chassis=FUJGECQ raid=raid0 branch=worker42


[gPanel] Installing OS on FUJGECQ as

[gPanel] Deploying DNS via Heaven…

hubot is deploying dns/master (deadbeef) to production.

hubot’s production deployment of dns/master (deadbeef) is done! (6s)

hubot is deploying puppet/worker42 (deadbeef) to production.

hubot’s production deployment of puppet/worker42 (deadbeef) is done! (6s)

[gPanel] booting Dell Inc. PowerEdge C5220 FUJGECQ (github-worker42-cp1-prd) into installing…

If we’re looking to spin up a number of hosts to expand capacity for a certain service tier, we can instruct gPanel to do this with our bulk-install command. This command takes app, role, chassis_type, and count parameters, selects the appropriate hardware from our ready pool, and initiates the installations.


/gpanel host bulk-install app=github role=worker chassis_type=C5220 count=3


[AC5] Dell Inc. PowerEdge C5220 JOYDSHP =>

[AD4] Dell Inc. PowerEdge C5220 NFDLTFH =>

[AD3] Dell Inc. PowerEdge C5220 FUJGECQ =>

hubot is deploying dns/master (deadbeef) to production.

hubot’s production deployment of dns/master (deadbeef) is done! (6s)

At this point gPanel will transition the chassis to our installing state and reboot the machine via IPMI. In this state we PXE boot the Ubuntu installer and retrieve a preseed configuration from gPanel. This configuration is rendered dynamically based on the hardware configuration and the options the user provided in their install command. Once the installation is complete, we move to the installed state where gPanel will instruct machines to boot from their local disk.

When we’re ready to decommission a host we simply tell Hubot, who will ask for confirmation in the form of a “magic word”.


/gpanel host destroy magic_word=orange



Deprovisioning chassis FUJGECQ.

Destroyed host Queued puppet cleaning job.

[gPanel] Deploying DNS via Heaven…

hubot is deploying dns/master (deadbeef) to production.

hubot’s production deployment of dns/master (deadbeef) is done! (6s)

[gPanel] booting Dell Inc. PowerEdge C5220 FUJGECQ (FUJGECQ) into ready…

gPanel transitions the chassis back to our ready state and makes it available again for future installations.


We’ve been pleased with the ease at which we’re able to bring new hardware into the data center and make it available to the rest of the company. We continue to find room for improvement and are constantly working to further automate the procurement and provisioning process.

Original URL:  

Original article

What’s new in curl

CURL keyboardWe just shipped our 150th public release of curl. On December 2, 2015.

curl 7.46.0

One hundred and fifty public releases done during almost 18 years makes a little more than 8 releases per year on average. In mid November 2015 we also surpassed 20,000 commits in the git source code repository.

With the constant and never-ending release train concept of just another release every 8 weeks that we’re using, no release is ever the grand big next release with lots of bells and whistles. Instead we just add a bunch of things, fix a bunch of bugs, release and then loop. With no fanfare and without any press-stopping marketing events.

So, instead of just looking at what was made in this last release, because you can check that out yourself in our changelog, I wanted to take a look at the last two years and have a moment to show you want we have done in this period. curl and libcurl are the sort of tool and library that people use for a long time and a large number of users have versions installed that are far older than two years and hey, now I’d like to tease you and tell you what can be yours if you take the step straight into the modern day curl or libcurl.


Before we dive into the real contents, let’s not fool ourselves and think that we managed these years and all these changes without the tireless efforts and contributions from hundreds of awesome hackers. Thank you everyone! I keep calling myself lead developer of curl but it truly would not not exist without all the help I get.

We keep getting a steady stream of new contributors and quality patches. Our problem is rather to review and receive the contributions in a timely manner. In a personal view, I would also like to just add that during these two last years I’ve had support from my awesome employer Mozilla that allows me to spend a part of my work hours on curl.

What happened the last 2 years in curl?

We released curl and libcurl 7.34.0 on December 17th 2013 (12 releases ago). What  did we do since then that could be worth mentioning? Well, a lot, and then I’m going to mostly skip the almost 900 bug fixes we did in this time.

Many security fixes

Almost half (18 out of 37) of the security vulnerabilities reported for our project were reported during the last two years. It may suggest a greater focus and more attention put on those details by users and developers. Security reports are a good thing, it means that we address and find problems. Yes it unfortunately also shows that we introduce security issues at times, but I consider that secondary, even if we of course also work on ways to make sure we’ll do this less in the future.

URL specific options: –next

A pretty major feature that was added to the command line tool without much bang or whistles. You can now add –next as a separator on the command line to “group” options for specific URLs. This allows you to run multiple different requests on URLs that still can re-use the same connection and so on. It opens up for lots of more fun and creative uses of curl and has in fact been requested on and off for the project’s entire life time!


There’s a new protocol version in town and during the last two years it was finalized and its RFC went public. curl and libcurl supports HTTP/2, although you need to explicitly ask for it to be used still.

HTTP/2 is binary, multiplexed, uses compressed headers and offers server push. Since the command line tool is still serially sending and receiving data, the multiplexing and server push features can right now only get fully utilized by applications that use libcurl directly.

HTTP/2 in curl is powered by the nghttp2 library and it requires a fairly new TLS library that supports the ALPN extension to be fully usable for HTTPS. Since the browsers only support HTTP/2 over HTTPS, most HTTP/2 in the wild so far is done over HTTPS.

We’ve gradually implemented and provided more and more HTTP/2 features.

Separate proxy headers

For a very long time, there was no way to tell curl which custom headers to use when talking to a proxy and which to use when talking to the server. You’d just add a custom header to the request. This was never good and we eventually made it possible to specify them separately and then after the security alert on the same thing, we made it the default behavior.

Option man pages

We’ve had two user surveys as we now try to make it an annual spring tradition for the project. To learn what people use, what people think, what people miss etc. Both surveys have told us users think our documentation needs improvement and there has since been an extra push towards improving the documentation to make it more accessible and more readable.

One way to do that, has been to introduce separate, stand-alone, versions of man pages for each and very libcurl option. For the functions curl_easy_setopt, curl_multi_setopt and curl_easy_getinfo. Right now, that means 278 new man pages that are easier to link directly to, easier to search for with Google etc and they are now written with more text and more details for each individual option. In total, we now host and maintain 351 individual man pages.

The boringssl / libressl saga

The Heartbleed incident of April 2014 was a direct reason for libressl being created as a new fork of OpenSSL and I believe it also helped BoringSSL to find even more motivation for its existence.

Subsequently, libcurl can be built to use either one of these three forks based on the same origin.  This is however not accomplished without some amount of agony.

SSLv3 is also disabled by default

The continued number of problems detected in SSLv3 finally made it too get disabled by default in curl (together with SSLv2 which has been disabled by default for a while already). Now users need to explicitly ask for it in case they need it, and in some cases the TLS libraries do not even support them anymore. You may need to build your own binary to get the support back.

Everyone should move up to TLS 1.2 as soon as possible. HTTP/2 also requires TLS 1.2 or later when used over HTTPS.

support for the SMB/CIFS protocol

For the first time in many years we’ve introduced support for a new protocol, using the SMB:// and SMBS:// schemes. Maybe not the most requested feature out there, but it is another network protocol for transfers…

code of conduct

Triggered by several bad examples in other projects, we merged a code of conduct document into our source tree without much of a discussion, because this is the way this project always worked. This just makes it clear to newbies and outsiders in case there would ever be any doubt. Plus it offers a clear text saying what’s acceptable or not in case we’d ever come to a point where that’s needed. We’ve never needed it so far in the project’s very long history.


Just a tiny change but more a symbol of the many small changes and advances we continue doing. The –data option that is used to specify what to POST to a server can take a leading ‘@’ symbol and then a file name, but that also makes it tricky to actually send a literal ‘@’ plus it makes scripts etc forced to make sure it doesn’t slip in one etc.

–data-raw was introduced to only accept a string to send, without any ability to read from a file and not using ‘@’ for anything. If you include a ‘@’ in that string, it will be sent verbatim.

attempting VTLS as a lib

We support eleven different TLS libraries in the curl project – that is probably more than all other transfer libraries in existence do. The way we do this is by providing an internal API for TLS backends, and we call that ‘vtls’.

In 2015 we started made an effort in trying to make that into its own sub project to allow other open source projects and tools to use it. We managed to find a few hackers from the wget project also interested and to participate. Unfortunately I didn’t feel I could put enough effort or time into it to drive it forward much and while there was some initial work done by others it soon was obvious it wouldn’t go anywhere and we pulled the plug.

The internal vtls glue remains fine though!

pull-requests on github

Not really a change in the code itself but still a change within the project. In March 2015 we changed our policy regarding pull-requests done on github. The effect has been a huge increase in number of pull-requests and a slight shift in activity away from the mailing list over to github more. I think it has made it easier for casual contributors to send enhancements to the project but I don’t have any hard facts backing this up (and I wouldn’t know how to measure this).

… as mentioned in the beginning, there have also been hundreds of smaller changes and bug fixes. What fun will you help us make reality in the next two years?

Original URL:  

Original article

Dave Chappelle Has Phone-Free Zone For His Shows Through Partnership With Tech Startup

dave chappelle Comedian Dave Chappelle seems to be fed up with people recording his shows and sharing his material on the Internet. Chappelle recently tapped tech startup Yondr, which makes cell phone cases that prevent people from using their phones, to help ensure audience members are more engaged during his performance at Thalia Hall in Chicago.
For Chappelle’s 13 sold-out shows at Thalia Hall… Read More

Original URL:  

Original article

A.I. and Automation… Po-tay-to, Po-tah-to

I am calling for an official moratorium on the term Artificial Intelligence in relation to the law!  Everyone please just stop using it. It’s a needlessly charged word that only confuses and clouds the underlying issues whenever it comes up.

From now on any time you feel the need to use the term Artificial Intelligence, replace it with Automation.  No seriously, they’re exactly the same thing, at least as far as the current legal market is concerned. Whereas, AI carries connotations of ‘robot lawyers’ replacing people, Automation seems friendly, simple, even mundane.  That’s good.  Automation is the future of legal practice.

My friend Ron Friedmann posted a Twitter poll last week that got my hackles up.

Which tech will have most impact on legal market in next 3 years?
[Reply to suggest other choices for next poll]

— ronfriedmann (@ronfriedmann) November 26, 2015

Come on people!  Really?  Collaboration software!?  Biggest impact on legal market in next 3 years? Do people even read the question before they start ticking boxes?

Don’t get me wrong, I am a huge fan of collaboration software.  I firmly believe that modern collaboration tools are a fundamental requirement for any modern law firm, akin to a document management system, a productivity suite, and maybe a handful of lawyers.  But the ‘most impact on legal market’?  Tech that has been widely available for 10 years, that everyone is already using, even if IT or firm management frowns on it.  I don’t think so.

The correct answer, and the one that was chosen by a majority of respondents, is Automation.  I know, Automation wasn’t officially a choice, but look at the options again.  AI/Machine Learning and Contract Analytics collectively received 58% of the votes. Contract Analytics is a form of AI/Machine Learning and they should have both been listed as Automation tools.

Woo hoo!  Ron’s readers aren’t dumb, they just got a little confused by the options. Easy to do, when the confounding term AI rears it’s ugly head.

This was all bouncing around in my head yesterday when I saw the following article on Bloomberg BNA.

Another Law Firm Adopts Automation Technology

In the latest sign that more and more legal services are being automated, Akerman has announced it will operate a data center that allows corporate clients to quickly look up data privacy and security regulations without having to consult a human lawyer.

Look at that. The beauty of it. The simplicity. The near total lack of hysteria about robots stealing jobs. And guess what words don’t even appear in the article:  Artificial and Intelligence.

But you know what that article is about?  The biggest impact on the legal market in the next 3 years.

Automation.  Or as I like to call it, the creation of Legal Engines, by Legal Engineers, to automate the practice of law one task at a time.

If only someone had foreseen that such a thing might happen.

Original URL:  

Original article

Wikipedia Creates AI System To Filter Out Bad Edits

An anonymous reader writes: Wikipedia has developed a new artificial intelligence system aimed at improving the quality of its entries and detecting both mistakes and damaging edits made to its articles. The technology is named the Objective Revision Evaluation Service. The Wikimedia blog explains that the system is able to highlight incorrect edits, allowing editors to filter them out from the “torrent” of new amends and scrutinize their credibility. The entire service and process is open – with Wikipedia making revision scoring transparent and audit-able by publishing the source code, performance statistics and project documentation publicly under open licenses.

Share on Google+

Read more of this story at Slashdot.

Original URL:  

Original article

How to Troubleshoot Windows 10 with Reliability Monitor

Troubleshooting problems in Windows can be kind of a pain, but the built-in Reliability Monitor gives you a leg up by presenting your computer’s reliability and problem history in one easy view.

Read more…

Original URL:  

Original article

Proudly powered by WordPress | Theme: Baskerville 2 by Anders Noren.

Up ↑

%d bloggers like this: