Devops (3): Chef (and some fun cooking)

After having looked into Vagrant, it became clear that Puppet and Chef are "the" tools to do the structured and repeatable configuration of machines. I picked the recent book "Learning Chef" to learn and experiment a bit.

Tools

• The book uses the recent Chef Development Kit which should gradually replace the tool called Knife.
• The tool "Chef" uses the the Recipe DSL to write recipes (Domain Specific Language, based on the Ruby programming language)
• VirtualBox is used for running the VM's that are going to be cooked and baked
• Complemented with our good friend Vagrant (see previous DevOps blog)(actually kitchen-vagrant)
• Finally the tool kitchen is used  

The focus op Chef really lies on getting the software on the machine installed, configured, and up and running. On the target machine, a Chef Client is installed that will retrieve (new and updated) recipes from a Chef Server. So the Chef clients each pull the recipes from the Chef Server.
Note: the tool Ansible does the opposite, pushing configuration data to the machines.

Many cookbooks or recipes can be downloaded from the Chef Supermarket and others.

Finally
After spending some time look around in the world of DevOps and its tools, I have a few general Observations:

• Devops is strongly focused on Linux
• Diverse programming languages such as Ruby and Erlang are used
• This is a domain in full flux, new initiatives and companies pop up, things evolve rapidly
• First time I see the file format YAML actually being used

Devops (2): Vagrant

After looking a bit into Docker, I went on to look into Vagrant, another well-known tool to provision and configure virtual machines.



To learn about Vagrant, I read the book "Vagrant: Up and Running". This O'Reilly book is well-written and the examples all work. While doing the exercises on a Windows Server 2012R2 VM, I hardly encountered any problems. Starting the Ubuntu VM with the vagrant up command and removing it with vagrant destory.

The Vagrant command line tool allows to create a Virtual Machine in a reproducible and neutral manner. Vagrant was initially developed around the the free VM product Oracle VirtualBox. But Vagrant now comes with many other providers, e.g. AWS, Rackspace, IBM SoftLayer and Microsoft Azure. Support for VMWare however is not free ($79).

Vagrant focuses on the creation of Virtual Machines in a neutral manner. Contrary to Docker, it uses an actual Virtualization solution to provision the virtual machines. This allows Vagrant to support multiple Operating Systems in parallel. And offers support for automating the creation of Windows based virtual machines.

When Vagrant is used in combination with Oracle VirtualBox, Vagrant will use the VBoxManage.exe of VirtualBox. To create machines with a cloud provider, the respective Vagrant provider will leverage the API and tools of the specific Infrastructure-As-A-Service solution. Vagrant configure all sorts of attributes of the virtual machine, incl. e.g. networking (and port forwarding).

For the actual provisioning of the machines, Vagrant supports many options, including command line. But most often, Vagrant will be used in combination with Chef or Puppet. E.g. the Chef development kit uses Vagrant as its default "driver".

Boxes
Vagrant does not start from an ISO image, but from an already prepared "box". The more such box is pre-configured, the fewer configuration needs to be done afterwards. Vagrant uses its own software format to package the virtual machines that are taken as a starting point (compare to Amazon Machine Images). Vagrantbox.es and many others make pre-packaged Vagrant boxes available,

Windows specific

• VagrantManager makes Vagrant accessible from the Windows (or iOS) Taskbar 
• The company modern.IE makes Windows boxes with all sorts of IE versions available.
• Interesting blog on how to create Variant Windows boxes
• Vagrant can directly access the command line of Linux boxes over SSH (secure shell). For Windows boxes this cal also be arranged wen cygwin (or other SSH server) is installed. But Vagrant can also use WinRM to access the Windows command line
• Where the installation of software on Linux boxes leverages apt-get or yum to install software packages, Chocolatery wants to bring a similar solution to the Microsoft world; many packages are available for quick and easy installation
• Boxstarter leverages Chocolatey packages to automate the installation of software and create repeatable, scripted Windows environments.

Vagrant and Integration Tools
In my own domain of Application Integration and SOA, I expect that both vendors and customers will pickup tools such as Vagrant for creating and provisioning (virtual) machines. Combined with Chef or Puppet to actually install and configure the software on these machines.

Devops and Docker

The holiday period between Christmas and New Year is an ideal period to catch up on some reading and experimenting. Devops and tools such as Docker, Vagrant, Chef, Puppet and Ansible were on my radar for a while. So finally some time to dive into this topics.

Nested VMs
Not to mess up my machine, I use VMWare workstation to spin up some test machines. As these Devops tools are all about creating and provisioning virtual machines, one must enable "Nested VMs" support. This allows one virtual machine to run in another.

Docker


Docker appeared on my radar while learning about Micro Services. Docker focuses on the creation of light-weight containers in which applications are configured in an automated manner.



The Linux Containers are very small by leveraging OS level virtualization of Linux. Is it some "chroot on rocks". The chroot system call on Unix/Linux changes the root directory for a program and all of its children. chroot allows programs - e.g. a web server - to run in a more protected mode. The OS level virtualization can limit all the resources used by child processes: CPU, memory, disk space, ... Because containers are so light-weight, many of them can be run on a single machine. This mechanism allows each application to run in its own container, its own virtualized OS.

To have a quick try of Docker, there is a great Online Tutorial consisting of 10 steps. Recommended!

As there aren't any books available on Docker, I watched the brand new training material of LiveLessons. As I couldn't find the text material, had to type over the instructions from the paused video. After wasting some time trying to get access the Fedora Atomic container on the Fedora 21 host, decided to switch to another topic, Vagrant. If I have some more time, I'll come back and retry with RHEL as used in the video training. Or switch to Windows and take a look boot2docker.

Micro Services - Conway Law and Application Integration teams

"Micro Services" is a new buzzword in world of IT architects. As it talks about application components communicating over a network and contains "services", it probably has something to do with SOA and integration. So I had to look into it.


Let's bulletize the description from the article by Martin Fowler and James Lewis:

• The Microservices architectural style is an approach to
• developing a single application < application architecture
• as a suite of small services, < componentization, no libraries
• each (service) running in its own process
• and communicating with lightweight mechanisms, < over the network
• often an HTTP resource API. < REST, message bus
• These services are built around business capabilities < Domain Driven Design
• and independently deployable by fully automated deployment machinery.
• There is a bare minimum of centralized management of these services, 
• which may be written in different programming languages and
• use different data storage technologies < eventual consistency

Microservices are an architecture used by very large, modern IT systems such as LinkedIn, Netflix, Amazon and eBay. There's all sorts of interesting aspects to Micro Services, e.g. the GUI part, security, transactional integrity, versioning etc.

Conway law - Integration Competence Center
But there was one aspect that triggered me in particular when learning about Microservices: Conway Law: "any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure".

So this law states that an application architecture will reflect the way an IT department is organized. Microservices advocates refer a lot to it.

Service boundaries reinforced by team boundaries (picture from article by Martin Fowler)

For Microservices to focus and align with business functionality, the teams developing (and maintaining) the Microservices should therefore be cross-functional, including the full range of skills required for the development: user-experience, database, and project management.

Orthogonal to the view of the Microservices architects, that Conway Law confirms my personal view and opinion that any IT organization that wishes to leverage a central integration platform to a great extent, requires a separate team developing on and managing that integration platform.


How did I learn about MicroServices?

• The Microservices article by Martin Fowler and James Lewis
• Rough cut version of the O'Reilly book "Building Microservices" by Sam Newman
• Software Engineering Radio podcast episode with Stefan Tilkov
• Software Engineering Radio podcast with James Lewis
• SAI evening session I attended with great presentation by Stijn van den Enden and colleagues

PS: when searching for the term "micro service", I found the term also in the book "Java Web Services Architecture" back from 2003!

ESB = Erroneous Spaghetti Box?

While re-reading the Microservices article by Martin Fowler, I was triggered by the following footnote #7: We can't resist mentioning Jim Webber's statement that ESB stands for "Egregious Spaghetti Box".  I viewed the presentation - from 2008 - in which Jim Webber and Martin Fowler bash the Enterprise Service Bus and translate the acronym ESB into Erroneous Spaghetti Box.

I do agree that often, the integration platform simply contains a spaghetti of point-2-point integrations. But that's good! Way better than all that integration logic dispersed over many systems. With a wide variety of integration techniques, protocols and message formats. And spaghetti in a box is exactly what I tell when explaining what an integration platform is. Only by taking the next step of careful service and message design, one can arrive at a true Service Oriented Architecture.

Let's sum up the main advantages of an integration platform:

• A standardized way to have applications talk to one another
• No coding in a 3GL such as Java or C# but configuration in an application specifically built for the task of integrating systems
• Support for applications of different kinds and ages, including packaged applications
• Strongly reduced diversity in the tools and techniques used to integrate applications
• Support for reliable, asynchronous communication using queuing and message persistence (which Fowler doesn't seem to like either)
• Trivial connectivity through adapters
• Central place to monitor and manage the communication between systems, in particular the reliable message exchange
• Help turn packaged or older applications into services if desired (not everything is developed in-house)

With the disadvantages:

• That it is a central, separate platform,
• Requiring some specific skills (XML)
• The cost of the integration development and support becoming truly visible.

Where Webber and Fowler do have a point, is that middleware vendors come with a whole slew of products. Obviously one should only pick the parts that are useful. And the ESB will definitely not create the Service Oriented Architecture for you.

Building blocks for government information exchange

Last Monday I taught the SAI workshop "Integratie: van EAI over SOA tot Cloud". I explained that there are many technical web services standards (WS-*). But that there is little attention from software vendors for the standardization of the functional side of web services. There are few standard XML schema building blocks to assemble the message exchanged via web services. XML building blocks are rather coming from B2B standards such as OAG, GS1 or UBL.

A workshop participant from the Flemish Government pointed me to the OSLO specification (in Dutch) they have created. Very interesting initiative where the Flemish government standardizes the building blocks for services. Allowing different government entities to exchange information: municipalities, Flemish government agencies and the Belgian federal departments.

In the introduction of the document, they well explain that web services focus on the standardisation of the box, but not on the nuts and bolts in the box. ("Het lijkt alsof je wel de dozen standaardiseert, maar niet de bouten en moeren die er in zitten").

To be fair, the OSLO spec doesn't go down to the actual service definitions either. It focuses on the definition of information blocks that can be assembled and used in (web) services. The spec. focuses on both XML/XSD and RDF. The XML schemas build further upon XML building blocks from UN/CEFACT and UBL

The workshop participant also explained me that the OSLO specification is strongly related to the ISA initiative of the European Union. "ISA is a program launched by the European Commission to improve electronic cooperation among public administrations in EU Member States". OSLO builds further upon the work of ISA.

Had never heard of the OSLO initiative nor the ISA program (while colleagues from Infront are actually involved). But that's the fun part of teaching, you learn so much.

BPMN reading material

As an IT consultant, one spends quite some time getting to and from your project locations. Lately I've been spending quite some time on the train. Ideal moment to refresh my BPMN knowledge. I picked the book "Real-Life BPMN" on Amazon Kindle. The authors - Jakob Freund and Bernd Rücker - both work for the German company Camunda, specializing in BPM(N) consulting and open source BPM engine.

The book is quite OK and shows lots of small examples to get acquainted with the BPMN notation. It is It start with a level 1 view on processes where a subset of the BPMN notation is used. Next it dives deeper with many more symbols (Level 2). One unique view expressed in the book is that the BPM models can be implemented with a BPM(N) workflow engine (level 3a), but also translated into an IT specification (level 3b) and implemented in code (level 4b).

A colleague of mine strongly recommended another book: "BPMN Method & Style" from Bruce Silver. And I must admit, the book is better structured and easier to read. Bruce Silver is really an authority in the world of BPMN.

 

Towards the end, both books well explain that BPMN 2 is not only a graphical representation but also an XML representation. Funny to see how Bruce Silver refers to BPMN XML examples from Camunda, the circle was round.

BPM podcast

As an IT consultant, one spends quite some time in his car. Preparing for a long drive to Germany, I went looking for some good podcasts. I recalled the German Software Engineering Radio.

The podcast I really enjoyed was Episode 198: Wil van der Aalst on Workflow Management Systems. Quite interesting to get a retrospective on where we are today with BPM systems. Recommended.

Background pictures on Windows 8.1

I've been searching for (royalty) free screen savers with art photos such as Magnum or Getty image. But haven't found any yet. During a meeting with Lhoist employees, I noticed the beautiful screen saver they had. The beautiful imagery was coming from the Lhoist collection and its screen savers. Something to be jealous of.

With my new high-res Windows 8.1 laptop (3200 x 1800), I also wanted to enjoy some nice pictures. For now I've settled for the daily images from Bing. After some experiments, I've settled for 2 applications to get them on my laptop.

First is Chameleon which I use to set the Windows lock screen.

Second app is Bing Desktop that I use for setting the desktop background.





Will stay on the outlook for ways to get art photos on the lock screen and desktop.