As I tore off the first month of my over-sized school-year calendar I was reminded of my first week at Puppet Labs. I woke up early, made breakfast, and caught a bus into downtown arriving 30 minutes early only to be reminded that I didn’t yet have a key to the office. I plopped down in front of the door to wait for someone else to arrive, and opened my notebook to review the topics for my first official meeting with my new boss. Scrawled in my notebook are limited, and partially illegible jottings I took from our previous conversation. Unfortunately, between his Australian accent, frequently altered speech tempo, and foreign use of commonplace terms, and the fact that my penmanship skills hover around a 3rd grade level, all I could make out is one line: “Puppet Camp-do it.”

With that extravagant amount initial information, the last four weeks, James’ guidance, and the help of our wonderful in-town event planner Julie, I have hobbled together what I think will be a pretty awesome event. So without further ado I give you the top five reasons to attend Puppet Camp:

Reason 1: You Own It

Puppet, and Puppet Camp are user driven. Without the community the product, and the event would fall to pieces. With that in mind Puppet Camp is run as an “unconference” or “open-space” conference. Essentially, while we have some morning lectures planned, they can be influenced by the audience and deviate from their topic or forget about it all together depending on your wants. Similarly our afternoon breakout sessions are completely user generated. During morning break and lunch you have the opportunity to suggest any number of topics for afternoon breakout sessions. Speaking of breaks of course brings us conveniently to…

Reason 2: The Edibles and Drinkables

We’ve got a pretty reasonably sized breakfast of fresh fruit, yogurt, bagels and cream cheese, and muffins to serve along with juice, coffee and tea. Breaks are accompanied by cookies, and soft drinks, and lunch features a soup, salad, roasted veggies, and a sandwich bar buffet with chicken, roast beef, and hummus. While you may be on your own for dinner, we are offering an open bar at Swig on Thursday night which will offer…

Reason 3: Networking Potential

Short of sharing a plate of spaghetti, there really isn’t a better way to forge relationships than over a pint and one of Swig’s 150 whiskeys and whiskies. Whether you want to find your next employer, meet Volcane, or just swap stories with other Puppet users, Puppet Camp provides the opportunity. Because of the events’ flexibility, it caters to novice Puppet users, Puppet Masters, and everything in between, behind, and around. You’ll be amazed by the people you’ll meet, the industries and companies they hail from, and the size of their Puppet implementations. As you converse, hopefully…

Reason 4: You’ll Learn, Learn, Learn

You’ll learn more about Puppet in 2 days that you could have in a month. You’ll learn about our future goals for the product, trade-secret work-arounds, and a whole host of other information. No matter how active you are in the community there is really no equal to learning, face to face, from other users. The exchange of knowledge Puppet Camp induces is by far its most valuable aspect, which leads me to want to end here but…

Reason 5: You Really Ought to Attend Developers Training

If you can stay the weekend you can sign-up to attend Puppet Developers Training starting on Monday, October 11th and ending Wednesday the 13th. Developers training teaches you how to extend Puppet by adding custom facts, types and providers, and more. This class is offered on a limited basis so be sure to sign up if you have the chance.

In addition to all opportunities and benefits listed above, you’ll get some exclusive Puppet Camp swag, and get a well deserved opportunity to visit San Francisco. You can reside with us at the Serrano Hotel using our group discount code or take to the city and explore. I hope you join us for Puppet Camp 2010. We’ve got an awesome line up of presenters and entertainment waiting for you in San Francisco.

What Happened and Why

Today we announced our Series B funding round. As most of you know, I’ve been at this for a long time – I started Puppet and (then Reductive Labs) Puppet Labs in March of 2005.  The plan was always to build a great infrastructure software company, and Puppet is one of the successful open source projects that’s always had commercial backing – this has been my full time job since that day in March.  My goal back then was to bootstrap, growing our revenues enough to continue hiring developers and increasing our product portfolio.  Over the years I’ve talked about products that I was really excited to build but never had the resources to, like Runnels (an infrastructure-focused message bus framework) and Nodify (recently released as the Dashboard).

Unfortunately, the reality I ran into is that it’s actually really hard to bootstrap a company like that, and over the last couple of years it became clear that fundraising was the best thing I could do to get the company moving faster.  We’ve been able to grow really well since our round with True Ventures last summer, and we’re finally getting a handle on our current development load, but our goals are still more aggressive than we’ve been able to accomplish.  We were on track to be profitable this year, but not to get all of the awesome things done that we want to work on.

This left us with either scaling back our ambitions, or scaling up our capital.  Given how the buzz around the cloud is driving interest in infrastructure, and especially in automation, it didn’t make sense to step into the background – now is the time to double down, not pass and hope to bet again later.  We don’t need a ton of money – we’re a lean company (good lessons learned from living deal to deal, where each new customer got me another quarter of being able to feed my family), and because we’re a software company our biggest expense is people, not servers or facilities.

As with the best investment deals, we’re almost as excited about the investors as we are about the money – Kleiner Perkins Caufield & Byers are just about the best there is out there, and Kevin Compton — who was already an investor (through his own firm, Radar Partners) but now will be joining the board — has a long and storied history of building and helping to build great companies.  I’ve been working with Kevin over the last year, and I’m entirely thrilled to be able to work even more closely with him, and to begin trying to understand this whole actually-running-a-business thing.

How does the News Affect our Users and Community?

You’ll see me out in the wild talking about why we took this money, but it all boils down to one thing:  Funding software development.  Most of that software will be open source, but some of it will definitely be commercial.  No, our investors aren’t making us do anything; this is our strategy and always has been, the investors are just helping us get it done.  We want to build a great company producing great software, and to do that we need to make money.  Turns out, selling software is actually a great way to make money.

I’ve never really thought of us as an open source software company, even though all of our software has been open source; I just think of us as a software company with a preference for open source.  We’ll be opening everything we can, but we also need to find a way to make enough money to pay the developers that produce that software, and we need to be able to fund speculative efforts and projects that take a long time to pay off, both of which are very hard when relying entirely on the services model that 100% open source usually dictates.

Note that we won’t be following an open core model – our open source software, including Puppet, will always stand on its own and be completely sufficient for nearly everyone.  There are, however, companies who have specialized needs, and we’ll be providing targeted applications at some of these companies.  In general, though, you’ll see us putting products out and trying out revenue models, and just generally experimenting and innovating in both software and business.  We think we have some great ideas, but we’re under no illusions that we know what we’re doing.

Get Involved

We hope everyone in the community will stick around for the next trip around the roller coaster, and we’ll be making sure we continue to spend some of our ill-gotten gains on increasing involvement and getting more user interaction.  You can always email me directly at luke@puppetlabs.com – and heck, you can usually just call me at +1-503-575-9774, although maybe not this crazy week. :)  And if you or someone you know is interested in moving out here to awesome Portland, OR and working with us to build the next generation data center stack, let us know.  We’ve got a few jobs posted now, but we’ll be adding quite a few more in the near future.

So we’re excited to announce that the 2.6 release is nearly ready and we’ll be putting out a beta release shortly.  It’s taken longer than we’d planned but the end result is packed full of new features and functionality.  We’re really excited about some of those features and where they’ll allow you to take Puppet.

As there are so many new features we’d like to begin to introduce you to some of the new hotness in 2.6 in the form of a series of introductory blog posts.  Posts written by the people who’ve been involved in producing and bringing those features to life.

We’ll be covering the replacement of the last XMLRPC components of Puppet with a full REST API and introduce you to that API.  We’ve got an exciting new construct called “run stages” that will better allow you to order and structure your Puppet runs, a new relationship syntax to make it faster and easier for you to specify the relationships between your  resources and the much anticipated introduction of hashes in the Puppet DSL.  We’ll also cover some cool additional language features that will make it easier for you to write your manifests – including our new Ruby DSL.

We’ve also done a lot of work in the core of Puppet to make it easier to work with your configuration data including enhancing how Puppet reports on what it does, adding more audit capabilities and helping you export information about your environment.

Finally, we’ve even redesigned the command line interface to allow you to use just the puppet binary to interact with all of Puppet’s functionality.

So look out for these upcoming posts and for the forthcoming release!

This is a big week in the world of Operations, perhaps the biggest week of the year, with the Velocity, Structure and DevOpsDays conferences all running.  Velocity, representing the ‘enfacts terribles’ of the WebOps world, and its more corporate cousin Structure address the needs of the Web and enterprise operations markets respectively.  DevOpsDays is the conference representing the nascent DevOps movement, which is trumpeting its message of cooperation and collaboration between development and operations teams.

The overall message of all these conference is that Operations matters.  Operations is the backbone of IT organisations and the group that carries the burden of ensuring that services are delivered to the appropriate service level and for the right cost.  This isn’t a surprise to most of us in the Operations world (and certainly not to the people who brought you Puppet!) but it’s heartening to see a broader understanding of this in the wider IT industry.

It’s exciting also to be involved in discussions about Operational culture, processes and tool stack and we’re particularly thrilled by the emergence of Configuration Management as the glue that holds together that stack (the ITIL folks are in the back saying “Yes, well we told you that ages ago!”).  The enterprise understands that merely having iron to run services and applications isn’t enough anymore.  Speed to market, fast fault resolution and better economies of scale in managing infrastructure require tools that understand what your environment should look like and provide the capabilities to make it so.  In the Cloud world too, both in hulking internal clouds and external Cloud providers, there is an acknowledgment that simply providing the computing power isn’t enough, as Opscode’s announcement of a SaaS configuration management tool reveals, you need to provide tools to provision and manage it.

Configuration Management though isn’t just needed in the enterprise and the Cloud.  Tools like Puppet manage infrastructure in the SMB world, where costs and resources are at a premium and every dollar expended in costs has a direct impact on the bottom line.  Puppet and other tools are deployed in education and government, in ISVs and ISPs, and to manage an increasing number of desktops, networks and embedded devices.  The penetration in these environments is also only just beginning.  Configuration Management and automation is a whole brave new world for many organisations and I look forward to seeing how it’s going to unfold.

We’ll be at all the conferences this week – Luke, Scott Campbell, and John Rutherford will be at Structure whilst I will be attending Velocity.  Luke and I will at DevOpsDays where we’re both speaking on panels about DevOps.  So if you’re also attending any of these events please look us up and have a chat about Puppet, Configuration Management or the state of Operations in general.

Many thanks to Patrick Debois for putting on a great event this year at Puppet Camp Europe. He has finished uploading the videos from all the speakers at the event to our Puppet Labs Blip.tv account.

I have embedded Luke’s keynote in this post below, but all the presentations are available at the following links:

• Portable infrastructure with Puppet – Luke Kanies (Slides|Video)
• Embedded Puppet – Alban Peignier (Slides|Video)
• Deploying datacenters with Puppet – A user’s perspective – Rafael Brito (Slides|Video)
• Puppet modules and module standards – Alessandro Franceschi (Slides|Video)
• Auditing change management policies with Puppet and Splunk – Jeff McCune (Slides|Video)
• Puppet in complex, real-time environments – Rick van der Zwet (Video)

Portable infrastructure with Puppet – Luke Kanies

It’s good to be back in the office again, pounding the 2.6.0 beta into shape, and wonderful to see my family, but I really had a blast at Puppet Camp EU as well.

The Belgians were very friendly, as were the Netherlanders I met the first day due to a little miscommunication on my travel plans. Only speaking two human languages myself, I’m constantly impressed by the linguistic fluency of the average European, who can offer to by you a beer in an impressive array of languages. (Luke claims that beer is the international currency of open source, and he’s right so far as it goes, but the Belgians, while justly proud of their brews also offer a reasonable exchange rate in cheese and chocolate).

Despite the hospitality I made it to Puppet Camp in time for Luke’s Keynote, which he structured around the promises we made at Puppet Camp SF and how we’ve progressed on them. The big announcement here was the Module Forge, our new public forum for sharing useful bits of Puppet code, which went live just a few hours before his talk. He also went through a list of features presently in the master branch which will be coming out in 2.6.0 (the successor to 0.25.x, bumped to 2.6.0 rather than 0.26.0 because of the consensus that we shot past the traditional 1.0 criteria without adjusting our version numbers), and talked about the new Dashboard and it’s features.

The most notable thing to me, though, was Luke’s conscious shift to only talking about the present in his keynote–as Yogi Berra famously may have said, predicting is hard, especially about the future. Discussion about the future of Puppet is just that, a discussion, involving lots of people, backed by lots of code experiments and empirical data, spurred on by creative insights and pulled forward by dedicated effort that goes on daily in our amazingly diverse and active community. Trying to predict the outcome of this process is a task best left to the amateurs. When I asked Luke if his new found reluctence to predict was going to be his policy from now on, he just smiled enigmatically.

After the keynote the real fun began with formal talks interspersed with the usual equally informative informal ones. Lots of ideas flying around, buzz about the changes coming with 2.6.0, talk about run stages, localization issues (always closer to the fore in Europe than in the mostly monolingual US), a general consensus that environments are confusing and need more love, lots of message buss goodness and people sidling up to the concept of provisioning from multiple directions, and MS windows support to name just a few of the many fascinating (to me at least) topics.

I picked up half a dozen bug reports and feature requests (mostly little things that hadn’t risen to the point of warranting a trip to puppet’s Redmine ticket system which, even though it’s always willing to hear from you, isn’t always handy when the thought strikes).

My favorite idea was an 80% solution to the issue of composite resource keys (things like “TCP:22″ where the identity of the resource depends on multiple logical properties of the resource). As David Schmitt pointed out, this turns out to be much more of an issue when dealing with non-posix file systems, where you run into things like “drive letters” and “host volumes.” We’ve long intended to support something like this but always got bogged down in the edge cases. As became apparent at Puppet Camp, there may be a simple solution that gets us 80% of the functionality for 20% of the effort. Fellow Puppet Labs developer Jesse Wolfe and I are going to work up a prototype in the next week or two.

If I had to pick one idea that resonated more than any, though, it would be the importance of running Puppet on the right side of the line between compliance enforcing and compliance violation. In many environments, having a known state is equally or more important than having the correct state. In environments where compliance with rules has liability or regulatory implications, making unauthorized “fixes” to a system can be a carrier ending move. In such environments, there are two basic paths a Puppet implementation can choose;

1. be a tool that runs in noop mode most of the time, with manifests carefully maintained to reflect system changes managed by some other process, lest the eventual puppet run introduce unintended reversions or other compliance violations
2. alternatively run Puppet aggressively as the compliance monitoring and enforcement mechanism and ensure that all changes in managed resources are made via Puppet (logging in a changing things on the fly not permitted)

In the former case, any change that Puppet makes runs the risk of being seen as breakage if it reverses some undocumented change made (perhaps months earlier) to fix a crisis. By definition, such a reversion runs the risk of reproducing the crisis, which is not a pleasant thought. Wherever possible, it’s far better to be the enforcer of compliance rather than the potential violator.

The main objection to this is the potential increase in downtime that can result if changes have to wait for the next scheduled Puppet run; unfortunately, the systems with the strictest compliance rules often have tight uptime bounds as well. Stephan Nellson-Smith described an interesting method of getting the best of both worlds by using git to manage the manifests and triggering puppet-runs locally to make changes that take effect in seconds but leave an audit trail and ensure that the state puppet is enforcing is always upto date.

Of course, there was much more to Puppet Camp than I can capture in a blog post. Deepak Giridharagopal did an amazing job of introducing the speakers, pumping energy into it at times when the rest of us we just getting started, and Patrick Debois’s prepwork paid off in the smooth flow. And, as always, Brice Figureau was delightful to chat with.

A few parting statistics from some of the many show-of-hands calls:

• about 10 out of 70 use publicly available modules, only 2 didn’t modify them
• 4 use Forman, 3 use Dashboard, 6 use ldap.
• ~30% have to deal with compliance monitoring
• ~10% using Splunk
• 100% care about finer grained notifys, requires, etc.

All in all this was a very productive show, both for the attendees and the Puppet Labs crew. I hope to see everyone at our next event.

Los Alamos National Laboratory has released a whitepaper on how they used Puppet to automate Mac OS X configuration management. The paper includes a walkthrough of their decision to use Puppet and detailed technical steps of their installation and configuration process.

View and download: Mac Configuration Management at LANL

If you are looking to use Puppet to automate Hadoop or HBase deployments you should check out this post linking to Adobe’s Puppet recipes that they have published on GitHub.

Adobe had released the following recipes:

• Creating the user under which the entire hstack runs.
• Changing system settings, like the ssh keys, authorizing machines to talk to each other, aliases for hadoop and hbase executables, /tmp rules.
• Standalone puppet module to deploy Hadoop
• Standalone puppet module to configure the Hadoop NameNode in High-Availability mode via DRBD, heartbeat and mon.
• Standalone puppet module to deploy HBase
• Standalone puppet module to deploy Zookeeper.

This is another great example of the strength of the Puppet community in addressing valuable deployment and configuration automation tasks.

Our Product Manager for Puppet, Michael DeHaan, joined Amanda Coolong and Mark Jeffrey this week on This Week in Cloud Computing. Check out the video below as they address Puppet as well as a variety of cloud computing topics.

Puppet Forge Will Be Used to Further Automate Common Configuration Management Automation Tasks

Puppet Labs, home of Puppet, the open source leader in data center automation, today announced the launch of Puppet Forge, a central repository for Puppet Modules. One of the key strengths of Puppet is its active and supportive community. Puppet Modules are sets of instructions used by Puppet to automate processes, whether it’s setting up a database server or web server. When Puppet users create these modules they can now add them to the Puppet Forge for the entire Puppet community to benefit from them.

A Puppet Module in Puppet Forge

Prior to the Puppet Forge users have hosted their own content for others to use on sites like github.com or on their own web servers. While this was valuable, it was inefficient since there was no central location to share or find new modules.

“Puppet Forge is a major step forward for Puppet Labs and the Puppet community. We want to make it as easy as possible to create, share and find Puppet Modules that can accelerate and enhance the value our user community gets out of Puppet,” said Luke Kanies, founder and CEO of Puppet Labs. “We are launching Puppet Forge to further enhance the automation of common configuration management tasks, but ultimately this just makes it easier for the community to benefit themselves and each other.”

Puppet Forge will be used to search for existing solutions to common IT configuration management problems. This now makes it possible to browse all of the community’s contributed content by category and share work created by some of the greatest minds in the Puppet community. This also provides a very easy mechanism for Puppet users to contribute content back to the community. They simply need to create an account and upload and categorize their Puppet Module.

“As Puppet Forge grows and gathers more content, we expect it will become the most popular way to find out how to deploy any application — this is the kind of sharing that open source is all about,” said Michael DeHaan, product manager at Puppet Labs.

Visit the Puppet Forge now