Applicability
This page deals with the Web-based components of the BMLT. There are other components, such as the iOS apps. These will be subjected to a similar regimen, but there are likely to be differences.
Basic Philosophy
The BMLT is treated as a “top shelf” software engineering project. The quality level is of utmost concern, and we need to remember that this project is being used by many people. It is not our private playground, and what we do with it can have tremendous impact on a lot of folks.
Like many aspects of NA Service, or infrastructure administration in general, when things go well, we won’t hear much. Things work the way they are supposed to work, and no one really takes much time to let us know things are OK.
However, when things go bad, it’s an entirely different story. Expect to hear from disgruntled users. Addicts, especially, can get a bit…“pithy” in their feedback.
With this in mind, we need to be very careful about the work we do, and the releases that we make. This should not be treated casually. We have signed on to a big Responsibility, and we need to make sure that we live up to it.
When preparing a release of the BMLT, there are a number of steps we need to take. These include testing, tagging, creating zipped downloads, and merging development codelines with release codelines.
In this page, we’ll cover the BMLT Satellite “Driver,” the BMLT Satellite Base Class, the four Satellites that derive from it (BMLT Basic, Drupal 6, Drupal 7 and WordPress Plugins), the Semantic Workshop, and the BMLT Root Server.
ENVIRONMENT
All of the above are kept in a set of Git repos (This is the main account for these repos on GitHub), with the exception of the WordPress Plugin, which uses SubVersion, and which is kept in the WordPress Repository. It is absolutely vital that all work be subjected to the strictest possible configuration management regimen. Quality is of utmost importance.
Here are each of the repos:
- The BMLT Satellite “Driver”
- The BMLT Satellite Base Class
- The WordPress Plugin (Here is our Git Master Repo)
- The BMLT Basic Satellite
- The Drupal Plugin
- The Semantic Workshop
- The BMLT Root Server (Just moved to GitHub)
Each repo consists of at least two branches: A “master” (or “trunk”) branch, and one or more “release” branches (with the exception of the Semantic Workshop, which just has a “Master” branch). The Drupal plugin has two release branches (one for Drupal 6, and another for Drupal 7).
The Master/Trunk Branch
The “master” branch is considered the project Mainline. This is always very stable, but has not been validated to release standards. Additionally, it may not be complete, for release targets.
The Trunk Test Server is usually kept to the Mainline version. The Stable Test Server, on the other hand, is kept at the Release version.
The Release Branch[es]
If code is in here, it needs to be “release quality.” That means that the code has been THOROUGHLY TESTED, and it contains NO EXPERIMENTAL OR DEBUG CODE. The debug mode flag in the PHP source should be commented out, and the changelist/commit should have a tag attached to it.
NO DIRECT MANIPULATION OF THE RELEASE CODELINE SHOULD BE DONE. The only exception is that the Drupal release branches may need to have an empty directory removed (more on that later). All Release branch commits should actually happen in the Master branch, and are integrated into the Release branch when we integrate over the Master branch.
Of course, the world isn’t perfect, and it’s inevitable that some minor tweak may need to be done in a release codeline. If that is the case, the changes should always be integrated back into the Master branch.
The Mainline Model
The BMLT is developed using the Mainline Model. This specifies that we don’t “branch branches.” Instead, we keep coming back to a main branch (the “master” branch in Git, and the “Trunk” branch in SVN). When we want to do new development, we use a “soft” branch, directly from the “master” branch. This “soft” branch is designed to be short-lived. It’s usually task-focused. When the task is complete, the branch is integrated (or, more often, copied, which means that it is a “hard” integration) back into the mainline, and the “soft” branch is discarded. WE NEVER RE-USE “SOFT” BRANCHES. “Soft” branches change a lot. In some cases, we may integrate back up to the mainline several times while working in the “soft” branch in order to make our work available to other users, or to make sure that our integration will be smooth.
In the Mainline Model, experimental branches are considered “soft,” and are “under” the mainline, while release branches are considered “firm,” and are represented as “above” the mainline.
Release branches are considered “firm.” They are long-lived, and rarely change, once they have been established. In our case, we tend to have a single branch that is updated infrequently (only at release time) from the master branch. Some developers prefer to establish a new release branch whenever a release is made. However, this is a redundant practice. Use tags on the release branch, instead.
The BMLT codeline model employs a reused, firm, tagged release branch, and a stable mainline, with occasional development branches. The yellow triangles are tags, the blue line is the Release Branch, the black line is the Master Branch (Mainline), the purple lines are development (temporary) branches. Red solid arrows are “hard” integrations, or “copies,” and the green dotted lines are normal integrations.
Because of the distributed nature of Git, we can work on an unstable mainline in our workstation repos, but we should not push our local repo to the origin repo (the ones linked above) until our mainline is stable.
However, we should endeavor to keep the origin mainline up to date with stable releases. As we develop in our local branch, we should push stable states back to the origin, so other developers will get what we are working on, and will have a chance to synchronize their own work.
On the other hand, we should also pull changes from the stable mainline if others are working on it, and ensure that we stay in sync.
It is important to do many small checkins and pushes, as opposed to single, big comprehensive ones.
RELEASE PROCESS
Pre-release Testing
Pre-release testing is done on the stable mainline. It is generally done on a local Git repo, but that repo should be synchronized with the origin mainline branch before testing begins.
We also have a testing server. This can be used to test SSL, as well as all of the satellite variants. The server can be updated manually from the mainline, so the test targets represent the release candidate code.
Testing and QA of the BMLT is a topic all to itself, and will be covered in a separate page.
Update the Version Number
Before the Mainline stabilization, we make sure that all appropriate version numbers (usually in file headers) are correct for the new release, and that the README.md file has the latest entry properly dated and complete with the changelist comments.
For the WordPress plugin, this means that the header for the readme.txt file needs to indicate that the “Stable Tag” attribute is set to the version of the new release (a tag that does not yet actually exist).
Mainline Stabilization
Once we are satisfied with testing, we stabilize the Mainline. This is done by checking in and pushing to the origin, all of the code deemed ready for release.
In the case of the WordPress plugin, this means creating a new directory in the “tags” branch. This directory is a copy of the new stable “trunk,” and is named to the version number (a simple numerical directory name). It is fastest to simply make a “bald” (no SVN information or Git information) copy of the new “trunk” directory, and check it into the server, as opposed to first checking in the trunk, then integrating over.
At this point, the Mainline is of release quality, and this is reflected in the origin server.
Updating the Release Branch
In Git, this is done fairly simply. You check out the Release branch, then integrate the Master branch into it. Push the new Release branch to the origin, and then tag the new version (the tag should be a simple version number). You may want to check the Master branch back out after doing this.
In the case of the WordPress plugin, we will check the Mainline into the “Trunk” branch. The WordPress system will scan this, and will eventually make it available to the users in the WordPress Plugin Repository. This is done automatically, once the Trunk has been pushed. Before checking it in, though, you need to make sure the “stable tag” line in the readme.txt file is set to the new version, and you need to upload the tag directory.
With WordPress, what you need to do, is take the current trunk, remove all the invisible .svn, .git and .gitmodule files (including inside the submodules) -I use a compressor program to do this, it automatically strips all that out-, and then upload this stripped directory into the “tags” directory, giving it the name of the tag (the raw version number).
Once this is done, then you can upload the trunk directory.
The WordPress system will send SVN changelist reports to the registered email for the “owner” account when changes are made to its repository. This is a security precaution, as they have had miscreants break into their system and modify plugins. This practice ensures that the author always knows when changes have been made to their registered plugins.
NOTE: We maintain the WordPress trunk directory in Git for convenience, but you need to use SVN to upload it to the WordPress repository.
The Drupal Module has 2 Release Branches: Drupal 6 and Drupal 7. The master Branch is integrated into each of these. In some cases, it may be necessary to remove an empty and unused module info file directory (an artifact of the Master Branch).
Tagging the Release
Once we have the release and master branches in sync, we create a tag. This tag will be reflected to the origin, and will consist of a very simple version number. That’s all.
Important Note About Submodules in Git
Git submodules are quite problematic. It’s important to make sure that your submodules are correctly applied to the Master branch AND the Release branch before checking them in and pushing them to the origin.
We use submodules for a very good structural reason. Each project needs its own repo, and using submodules makes the relationships between these repos clear and simple. Clear and simple is very important for an infrastructure service like the BMLT. It’s a VERY bad idea to assume that the user of the repo is familiar with package managers like Repo.
Just because you are an überGeek, doesn’t mean everyone else is.
In any case, the BMLT Satellite Base Class is a submodule that is used in virtually every project. In turn, this subproject includes the BMLT Satellite Driver as a submodule.
The BMLT Semantic Workshop is a submodule that is included in the BMLT Root Server project.
In order to ensure that Git submodules are up to date, you should run the following command line in the Git repos that include submodules (pretty much all of them):
$> git submodule update --init --recursive
However, if you use a GUI client, like Atlassian SourceTree, it may be a simpler matter.
In any case, it is very important to ensure that the submodules are up to date BEFORE pushing to the origin.
Version Numbering
It’s important to keep version numbers up to date. If you make a new release, it should have a new version number.
Version numbers usually appear in the headers of source files, as well as in a changelist comment in the README.md (or readme.txt) file. The README format is done in Markdown. Please keep the format of comments in these files consistent.
The format for version numbers is
; where
represents major functionality/user interface changes (such as addition of significant new functionality or new GUI that requires new documentation),
represents things like minor tweaks to GUI or functionality, but no real deviation from the documented functionality, and
represents bug fix versions, where there’s no visible UI changes.
Choosing which version to pick can be a bit arbitrary, but we should use discretion when we select a new version.
Git tags should be assigned only the version number (such as ”
” or ”
“).
When saving a WordPress tag, use only the numerical (separated by periods) version number as the directory name.
For the satellite projects, we keep all the release versions the same, and that should be based on the version of the BMLT Satellite Base Class (the BMLT Satellite Base Class contains about 95% of the code used for all satellites, so most of the work in releases goes there, and is propagated out to all the satellites at once). In rare instances, we may need to release an individual satellite with a fix that does not propagate to other satellites. In this case, the next BMLT Satellite Base Class should take the next version past the individual satellite that was changed, and that version will go out to all the satellites at once (For example, say the main satellite release was version 3.2.4, but a bug was found and fixed in the WordPress satellite, so it was increased to 3.2.5, while the other satellites would remain at 3.2.4. The following release of the BMLT Satellite Base Class would be 3.2.6, which would propagate to all the satellites).
It’s very important to keep the version numbers in all the files up to date.
Don’t Tinker
It’s important not to make too many releases. That means that you shouldn’t “tinker,” where you “play around with things,” or apply a little “eye candy.” If you want to do that, fine; but don’t make a new release with your experimenting. Add it in the working Master branch, and include it in the next major release.
Remember that each release is often downloaded by many folks, and applied in many places. It may be a burden to expect them to keep updating, just to get your “playing around.”
As it is, the general pattern tends to be that we make a major release, inevitably, bugs are found fairly quickly, and the release is followed by a series of smaller bug fix releases. Eventually, it stabilizes.
Until the next major release.
Always Maintain A Detailed and Complete README
It’s very important to be complete and detailed in the README changelists. Even if it’s embarrassing, add it in there. This is an open-source, version-controlled project. They’ll find out anyway.
PUT EVERY SINGLE CHANGE YOU MAKE INTO THE README. In my case, I generally update the README of the project as I go along. The lions’ share of the work I do is in the BMLT Satellite Base Class, so I will copy that README and paste the relevant release notes into the satellite projects that consume the base class.
We use the GitHub Markdown Syntax for our README. The README is called “README.md”, as that will be automatically parsed by BitBucket.
While a new version is being developed in the master (or softer) branch, I’ll mark the version and a release date of “TBD” in the README. Make sure you follow that whacky syntax of asterisks and dashes. That makes sure the dates and version headers are displayed correctly.
The dependency of the various projects is shown in this diagram. First, the BMLT Satellite “Driver” is released, then the BMLT Satellite Base Class, after which the Satellites are released.
Meanwhile, the Semantic Workshop is released, and it and the BMLT Satellite Base Class are integrated into the BMLT Root Server.
Cascading Release
When we make a new release, it is usually done in a “cascading” fashion, with certain projects being released before other projects.
The general order of release would be:
- The BMLT Satellite Driver
- The BMLT Satellite Base Class
- All of the Satellites (Order Doesn’t Matter)
- BMLT-Basic
- WordPress
- Drupal 6
- Drupal 7
- The BMLT Semantic Workshop
- The BMLT Root Server
For the Satellites, most of the work is done in the BMLT Satellite Base Class, so that is usually where the release cascade begins.
Remember that the BMLT Root Server also incorporates the Base Class, but in a somewhat peripheral fashion. The Base class is absolutely necessary for all of the satellites. In the Root Server, it is only used in the Root Server Browser.
The Root Server also incorporates the Semantic Workshop. Again, this is a somewhat peripheral functionality. It is not crucial to the main Root Server operation.
This means that work done in the BMLT Satellite Base Class will require that all the satellites, as well as the Root Server, be released in order to apply the fix/new functionality. The urgency of these releases should be left to your discretion.
Compressed Binaries
Once the source code has been committed and the tag set, we should bundle up a .zip (not .tar or .gz) of the working directory. THIS .ZIP SHOULD NOT HAVE THE .GIT, .GITMODULES or .SVN COMPONENTS IN IT. This is very important. We don’t want to pollute users’ directories with invisible cruft.
These binaries are uploaded to the “Downloads” section of each BitBucket repo. Care should be taken to match the name of the existing binary exactly (watch out for case-sensitivity). Delete the old one first, then upload the new one. This is important, because BitBucket will assign a different link to the file if it is uploaded before the old one is deleted.
CAUTION
We need to remember that the origin repos are public repos (this is a 100% open-source project, after all). It is very important that we NOT INCLUDE CONFIDENTIAL INFORMATION. This is things like our
file, or other files (especially invisible ones like
). These files could contain information that would be VERY BAD if released to the public. Since these are Git (or SVN) repos, they will stay available forever, even if we delete them.
CHECK CAREFULLY BEFORE CHECKING IN. Remember that if it is in your local repo, it will end up in the origin, as well, once you push. This includes files that you checked in, then deleted. That history remains in the repo, and is sent to the origin when you push.
Tweeting Releases
When we make new releases, we should tweet the release, along with a link to where the release should be fetched, using the official @BMLT_NA account. There is no need for detailed explanations. The README should have that.
- Other Topics
- An Illustration of the BMLT System
- BMLT Roles
- Is Your Server Set Up Correctly?
- Test Site Links
- The BMLT Satellite Base Class
- The BMLTiOSLib iOS Framework
- Configuration Management And Release