Genesis

Genesis is an application provisioning tool used to automate the creation of software and configuration packages. It is designed to simplify the delivery of large scale provisioning and deployment projects, with the objective of significantly reducing the time and money invested in them.

Design Principles

  • Delivers an intuitive solution that is user friendly.
  • Enforces your standards and specifications using XML Schema and regular expressions.
  • Provisions high quality application builds across the entire enterprise stack.
  • Create custom software and configuration packages for applications and deploy on target platforms in seconds.

Key Features and Benefits

  • Automates software and configuration package creation for all types of packages.
  • Standardises the provisioning of applications across the enterprise.
  • Validates application configurations during build, post installation and runtime.
  • Implements, enforces and verifies internal standards such as nomenclature.
  • Reduces project lead times and costs.

News

Genesis Code checked into SourceForge - Sun, 18 May 2008
The first cut of the Genesis code has been checked into SourceForge. We have now moved our source repository from dev2dev's codeshare site to SourceForge and migration of all Genesis artifacts (code, documentation and website) is now complete. (0 comments)

Documentation

We are constantly updating our documentation so please check these pages regularly.

How To Guides

Getting Started for Linux OS
Getting Started for Windows OS
Developers Guide
MyApplication Configuration Package

Technical Architecture

Engine and Core API
Specification Documents
Configuration Hierarchy
Archetypes
Schemas

FAQ

Coming Soon…

Presentations

Coming Soon…

Talk Back

Genesis is a made available by the efforts of our members. While we try to attract as much sponsorship as possible the project remains independent and is community driven. We encourage active participation by individuals or groups who wish to contribute positively to the project. All support and involvement is most welcome.

Download, Install and Feedback.

  • Download Genesis - The best way to help the project is to use Genesis and provide feedback. You can download source or binary packages.
  • Report Bugs - No project is perfect, and we need your feedback on any and all issues you find with the software.

Contribute Code or Documentation Patches.

  • Review The Documentation - Please make sure that you’re reviewed our online forums and documentation many questions will be answered in our FAQ . This way you can have a solid knowledge of the components of the project and avoid duplicating any existing efforts.
  • Download Source Code - Getting the source to Genesis is easy. Simply download the latest source or binary tarballs located on Source Forge.
  • Contribute Patches - Creating patches is probably the best thing you can do to contribute to moving the project forward. Just make sure that you’ve checked the forums and mailing list before you start to make sure you aren’t duplicating any existing efforts.

Participate in the Community.

  • Online Forums - The online forums for Genesis should be your first stop in researching an issue or asking a question.
  • Mailing List - When you can’t find an answer in the forums, you may want to consider asking your question on the appropriate mailing list . Separate list are available for users, and developers.

Archetypes

Software and configuration packages come in many different flavours form source packages like Red Hats SRPMS to binary packages such as Suns Solaris PKG and Microsofts MSI. Configuration packages like those created with Debian’s config-package-dev utility and other application archives such as WAR’s and EAR’s bear testimony to proliferation and variety of package types. With this in mind its easy to see the value of a single tool and approach to create all these types of archives.

Genesis works on some basic assumptions that allows it create these types of packages. Namely that these archives generally code some of the file types listed below:

  1. Source Code - Software sources.
  2. Binaries - Compiled source code.
  3. Build Instructions - build.xml files.
  4. Configuration - The software’s logical set-up.
  5. Content - External data provided to the application to use, mange or serve.

An archetype is a template of an applications configuration files. Generally any non binary or non content application artefact will be included in an application archetype. Application archetypes implement the designs, standards and specifications defined in an organisations standard build for the application and Genesis maintains archetypes for all the applications that exist in an enterprise stack.

To create a deployable software of configuration package, Genesis draws custom configuration data and combines it with the application archetype in a process called transcription. Since the same application archetype is used to build the packages for different environments there will be a high degree of standardisation across all the deployed instances of the application.

In order to maintain standardisation across the enterprise the best practice is to maintain one archetype for any single version of an application. In reality this may not be achievable, and you may find that you need to maintain two or three archetypes per application, however the more archetypes you use for a given application the less standardised the deployment of that application becomes. Under these circumstances you should review the causes of variation and make sure they are justified.

Transcription

The process by which configuration data is combined with the application archetype is called transcription. Transcription produces an application configuration package (e.g. RPM, Tar ball, Zip file) of the application that can deployed by third party orchestration tools such as those provided by HP or BMC. The transcription process also conducts compliance testing on the package to ensure it is configured in accordance with organisational standards.

Genesis Archetype Product Catalogue

Genesis application archetypes are maintained as a product catalogue. Each archetype comes with a standard build document that details it’s design decisions, standards and specifications used to create it.

Configuration Hierarchy

Every configurable hardware or software entity in a provisioning and deployment project is treated as an application in Genesis. That is to say that the configuration data and meta data used to build and install software, ranging from server operating systems to network device configurations, is created using the same process. These configurations are maintained in a series of flat files which are formatted in XML that conforms to the Genesis schema. The Genesis configuration hierarchy differentiates between logical and physical configurations and branches them appropriately.

Logical Configuration

The logical branch of a configuration describes the services, environments and applications deployed by the enterprise. Within every environment multiple applications may be used to implement a particular service. For example ACME Bank may have an on-line banking service deployed to development, UAT, integration and production environments. In the development environment a WebLogic server and Oracle database are used to implement the service, while on the production environment a complete suite of applications might be deployed.

Physical Configuration

The physical branch of a configuration describes the properties of a real objects. ACME banks physical configuration branch might describe multiple data centres with numerous racks installed with dozens of servers and network devices. When Genesis builds an application it combines logical and physical configurations and transcribes them with the application archetype. The result is an application that is configured for a logical environment located in a physical data centre.

Getting Started for Linux OS

The following instructions describe the steps involved in getting the Genesis demonstration application installed and operational. We used a server running Fedora 8 for this installation but in practice any GNU/Linux distribution is suitable. Genesis will also run under windows but this guide does not describe that installation. If you already have Java and Ant installed you may skip to section 4. Environment Set-up.

1. System Requirements

Genesis requires a Java Development Kit 1.6 and Apache Ant 1.7.0. You can download them from the links listed below.
Java Development Kit
Apache Ant
Genesis

2. Installing Java

Sun offers of choice of Java runtime packages for download. In this example we downloaded the jdk-6u6-linux-i586.bin file to our /tmp directory. After you have downloaded the file you should move it to a reasonable location and extract it. We moved the package to the /opt folder, made it executable and extracted it there. You should do this as the root user.

# Become root user

[user]$ su -

# Move the package from /tmp to /opt

[root]# mv /tmp/jdk-6u6-linux-i586.bin /opt

# Make the package executable

[root]# chmod +x /opt/jdk-6u6-linux-i586.bin

# Extract the package

[root]# /opt/jdk-6u6-linux-i586.bin

# Create a symbolic link to the jdk

[root]# ln -s /opt/jdk1.6.0_06/ jdk

3. Installing Ant

The Apache Foundation provide a number of archive formats for Ant. In this example we used apache-ant-1.7.0-bin.tar.gz. After we had downloaded it we moved the archive to the /opt folder from /tmp and extracted it there. You should do this as the root user.

# Become root user

[user]$ su -

# Move the package from /tmp to /opt

[root]# mv /tmp/apache-ant-1.7.0-bin.tar.gz /opt

# Extract the package

[root]# tar zxvf /opt/apache-ant-1.7.0-bin.tar.gz

# Create a symbolic link to the jre

[root]# ln -s /opt/apache-ant-1.7.0 apache-ant

4. Environment Set-up

Before installing genesis we need to set up our environment to make sure Genesis has everything it needs. This entails setting up a genesis user and making sure the environment variables for JAVA_HOME and ANT_HOME are set-up correctly and exported.

# Become root user

[user]# su -

# Add a genesis user and group

[root]# useradd -u 1010 genesis

# Choose a secure password for this user

[root]# passwd genesis

# Become the genesis user

[root]# su - genesis

Edit the genesis users .bash_profile and add the JAVA_HOME and ANT_HOME environment variables. see the example .bash_profile below:

###########################################################

# .bash_profile
# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi
# User specific environment and startup programs
JAVA_HOME=/opt/jdk

ANT_HOME=/opt/apache-ant
PATH=$PATH:$HOME/bin/:$JAVA_HOME/bin:$ANT_HOME
export PATH JAVA_HOME ANT_HOME

###########################################################

# Confirm the environment variables are correct:

[genesis]$ source .bash_profile

[genesis]$ echo $JAVA_HOME

/opt/jdk

[genesis]$ echo $ANT_HOME

/opt/apache-ant

# Confirm Java and Ant versions

[genesis]$ java -fullversion

java full version "1.6.0_05-b13"

[genesis]$ ant -version

Apache Ant version 1.7.0 compiled on December 13 2006

5. Genesis Installation

Once we have installed Java, Ant and our environment has been configured we are ready to install Genesis. We need to download it and extract it to a suitable directory.

# Become genesis user

[user]$ su - genesis

# Make a dist directory in the genesis users home

[genesis]$ mkdir dist

# Move the package from /tmp to /home/genesis/dist

[genesis]$ mv /tmp/genesis-{version}.tar.gz /home/genesis/dist

# Extract the package

[genesis]$ tar zxvf /home/genesis/dist/genesis-{version}.tar.gz

# Change directory to the genesis distribution

[genesis]$ cd /home/genesis/dist/genesis-{version}

# Execute the genesis without any options to see usage.

[genesis]$./genesis

Schemas

XML schema’s have a number of uses in the XML world, these include validation, documentation, querying support, data binding and guided editing of XML files. Within Genesis schema’s fulfill two primary functions, these are:

  • Validation of data stored in the configuration hierarchy.
  • Enforcement of infrastructure nomenclature.

Apart from keeping the Genesis engine free of validation code, leveraging XML schema for these purposes provides a robust, extensible and open standards compliant mechanism for ensuring the integrity of configuration hierarchy.

All Genesis XML Schema files (.xsd files) are stored under genesis/engine/schemas within the genesis distribution.

Configuration Hierarchy Validation

In order to maintain the integrity of the data stored in the configuration hierarchy Genesis schemas exist for all elements of the logical and physical configuration and are named after the elements they validate. Every application being provisioned by Genesis will also have it’s own schema. The schemas use a combination of predefined simple data types and derived simple and complex data types. These schemas maintain and enforce a standard and consistent structure and semantic for all the XML files stored in the configuration hierarchy.

Enforcement of Nomenclature

The role of nomenclature in the automation and standardisation of infrastructure provisioning cannot be overstated. A simple, consistent and justifiable infrastructure nomenclature is the foundation which all automated processes are built upon, yet the benefits of such a nomenclature reach far beyond provisioning. Encoding and validating naming rules has traditionally been a time consuming and expensive activity which has limited the extent to which the rules are can be applied and enforced.

Genesis enforces infrastructure nomenclature by encoding the naming rules using regular expressions to define simple data types within XML schemas. This mechanism is incredibly powerful, new naming rules can be quickly implemented, existing rules can be changed and “what if” scenarios can be tested with ease against existing configurations.

The following schema files are compose validate and enforce nomenclature in the configuration hierarchy:

Global Configurations

common.xsd
enterprise.xsd

Logical Configurations

service.xsd
environment.xsd

Physical Configurations

datacentre.xsd
rack.xsd
node.xsd

Download

Genesis Binaries

Download from Source Forge

View the getting started guide for Genesis installation instructions.

Genesis File Releases

Please read the release notes for the most up to date information about the package.

genesis genesis-0.0.34-tar.gz released (Sun, 04 Jan 2009 12:31:09 GMT) - Sun, 04 Jan 2009

Released at Sun, 04 Jan 2009 12:31:09 GMT by salimb
Includes files: genesis-dist-0.0.34.tar.gz (109449 bytes, 70 downloads to date)
[Download] [Release Notes]

genesis genesis-0.0.33.tar.gz released (Sun, 12 Oct 2008 16:49:55 GMT) - Sun, 12 Oct 2008

Released at Sun, 12 Oct 2008 16:49:55 GMT by salimb
Includes files: genesis-0.0.33.tar.gz (206680 bytes, 28 downloads to date)
[Download] [Release Notes]

Specification Documents

Overview

These documents detail the design and configuration choices used for enterprise, environment and application specifications. The examples provided cover only basic implementations but serve a good example for starting your own specifications.

Application Specifications

Developers Guide

The following instructions describe the steps involved in building a Genesis distribution from source. We used a server running Fedora 8 for this installation but in practice any GNU/Linux distribution is suitable. Genesis will also run under windows but this guide does not describe that installation.

1. Getting Started

The Genesis Source uses Maven 2 (http://maven.apache.org) for generating a usable Genesis installation. Note however that Maven is not required to use Genesis. To install Maven, follow the instructions available from the apache software foundation. The Genesis source is arranged into a number of modules. Currently, these modules are:

GENESIS_SRC_ROOT/
ant-tasks/
dist/
model/
schemas/

ANT-TASKS
This module contains the ant tasks for genesis such as those for resolving genesis objects to file names, building filter sets, and iterating of Genesis objects.

DIST
This module contains all the sample files packaged into the distribution, along with all of the packaging configuration to produce the genesis-dist.zip file in the correct layout.

MODEL
This module contains all the code necessary to work with genesis.xml configuration files. It defines the underlying object model for Genesis, along with the mechanisms to load and query them.

SCHEMAS
This module contains the definitions for the XML schemas relevant to the genesis core, such as the schema for the genesis.xml file.

2. Building a Distribution

To generate a distributable Genesis binary, execute:

mvn package

You must be in the GENESIS_SRC_ROOT directory. This will download all necessary dependencies, and build each of the modules. The output distributable can be found in the dist/target directory as a .zip file.

Engine and Core API

Overview

Written in Java, the Genesis engine and its core API is one of three key architectural components together with the configuration hierarchy and application archetypes.

The Genesis engine drives the creation of packages by using application archetypes. Archetypes, in their simplest incarnation, contain two files which detail the build instructions and the associated configuration data for a particular application. The build instructions are written in XML whilst the configuration file contains tokens which are replaced by values obtained from the configuration hierarchy. The resulting output files are copied to the build directory where they are packaged and ready deployment.

The process of creating archetypes and populating the configuration hierarchy is simple and intuitive. The Genesis engine is built upon and extends Apache Ant. This has number benefits for Genesis users:

  • Apache Ant is a ubiquitous build tool amongst the Java community and is used to compile and package Java code. By building the Genesis engine on top of Ant we ensure degree of familiarity with Genesis amongst this community and an excellent level of technical support.
  • The Genesis engine can be extended using the same mechanisms as are available to Ant. This simply entails writing Ant tasks which are written in Java.
  • All Apache Ant tasks and extensions are automatically available to Genesis.

During the creation of an application package, Genesis can execute defined tests to ensure the application configuration meets defined standards. For example, Genesis can check the validity of security certificates used by an application or test for correct networking configuration.

Getting Started for Windows OS

The following instructions describe the steps involved in getting the Genesis demonstration application installed and operational for Windows OS. If you already have Java and Ant installed you may skip to section 4. Environment Set-up.

1. System Requirements

Genesis requires a Java Development Kit 1.6 and Apache Ant 1.7.0. You can download them from the links listed below.

Java Development Kit

Apache Ant

Genesis

2. Installing Java

  1. Download the jdk-6u7-windows-i586-p.exe installation files.
  2. Double click the application package and Window’s Installer will begin the installation. For convenience we installed toC:\jdk
  3. Test the installation by typing javac at the prompt.

3. Installing Ant

  1. Download the apache-ant-1.7.1-bin.zip from the Apache Ant Downloads page.
  2. Extract to a suitable directory, e.g. C:\ant
  3. The installation can be tested by navigating to the C:\ant directory and typing ant at the prompt; the message should read:

    Buildfile: build.xml does not exist!

    Build failed

    This indicates a successful installation. Typing ant -version gives the version and the date it was compiled:

    Apache Ant version 1.7.1 compiled on July 17 2008

4. Environment Set-up

Before installing genesis we need to set up our environment to make sure Genesis has everything it needs. This involves setting the correct Environment Variables for JAVA_HOME and ANT_HOME. Environment Variables are edited via:

Start> My Computer> View System Information> Advanced> Environment Variables

  1. Add the New User Variables ANT_HOME set to C:\ant and JAVA_HOME set to C:\jdk (assuming the JDK root is C:\jdk and the Ant root is C:\ant).
  2. Add a New User Variable with the variable name PATH and the Variable Value C:\jdk\bin;.;C:\ant\bin This will show Windows where the Java compiler and the Ant binaries are.

5. Installing Genesis

Once we have installed Java, Ant, and our environment has been configured we are ready to install Genesis. The Genesis genesis-{version}.tar.gz files can now be downloaded from the homepage and extracted to a suitable directory (e.g. C:\genesis-{version}). To test the installation navigate to the genesis root and execute the genesis.cmd to show usage.

MyApplication Configuration Package

The Archetype

We can view the test archetype for MyApplication in the genesis-{version}

genesisarchetypesMyApplication directory. The archetype appears as follows:

# Static properties
CommonProperty=@Common.Name@
EnterpriseProperty=@Enterprise.Name@

# Logical Properties
ServiceProperty=@Service.Name@
EnvironmentProperty=@Environment.Name@
ApplicationProperty=@Application.Name@

# Physical Properties
DatacentreProperty=@Datacentre.Name@
RackProperty=@Rack.Name@
ServerProperty=@Server.Name@

The Build

Open the command prompt and navigate to the Genesis-{version}genesis directory. Run the build tool, this time specifying the archetype property values:

genesis.cmd MyEnterprise MyDatacentre MyRack MyServer MyService MyEnvironment MyApplication

The Application Configuration Package

A successful build creates a new directory in the Genesis root called build with the following directory structure:

buildMyEnterpriseMyServiceMyEnvironmentMyDatacentreMyRackMyServerMyApplication

The new MyApplication.cfg file which holds the Static, Logical, and Physical properties specified for the application configuration package is copied to the MyApplication folder, and appears as follows:

# Static properties
CommonProperty=CommonProperty
EnterpriseProperty=MyEnterprise

# Logical Properties
ServiceProperty=MyService
EnvironmentProperty=MyEnvironment
ApplicationProperty=MyApplication

# Physical Properties
DatacentreProperty=MyDatacentre
RackProperty=MyRack
ServerProperty=MyNode