Who’s Actually Using Bitbucket?

I recently had to deal with an Bitbucket user issue where people were being added to a JIRA group automatically that granted r/w access to Bitbucket. Nearly half of the people who were in this group had never even logged into Bitbucket, but they counted against our license seats. We were running dangerously close to running out of seats and had only hours to spare. Rather than focus on the problems that lead up to this situation, I’d rather focus on the core question “Who’s actually using Bitbucket?” and how we regained nearly 1000 seats.

Problem #1: How many people are actually using Bitbucket?

In order to cast a wide enough net to find everyone, our metric was “Who’s ever logged into Bitbucket?” To do that, we queried the database (MySQL) and exported the results to CSV. Below is the query and what the resulting output looked like.

select cua.attribute_value, cu.user_name from cwd_user_attribute as cua
inner join cwd_user as cu on cu.id = cua.user_id
where cua.attribute_name = "lastAuthenticationTimestamp";

The resulting CSV looked something like this:

attribute_value,user_name
1497642118994,user_a
1497642069440,user_b
1497642117652,user_c

Problem #2: How do we put these people in a new permissions group?

To solve that, we turned to JIRAs REST APIs. First we created a new group in JIRA, then we wrote a little bit of code to loop through the CSV and add those folks to the group. I need to preface that this snippet was written very hurriedly to solve the problem (I have an ego to protect). This is written in Python 3 and uses the Requests module.

import requests
import csv

url = "https://<JIRA-URL>/rest/api/2/group/user?groupname=group-name"
admin_user = "admin"
admin_pass = "password"

headers = {'content-type': 'application/json'}
post_data = """{"name": "%s"}"""

def make_json(user):
 return(post_data % user)

with open('active_users.csv', newline='') as csv_file:
 reader = csv.reader(csv_file)
 for row in reader:
 user = row[1]
 response = requests.post(url, auth=(admin_user, admin_pass), headers=headers, data=make_json(user))
 if response.status_code == 200 or response.status_code == 201:
 print('{} added to new group'.format(user))
 else:
 print('Failed to add {} to new group'.format(user))
 print('Status code: {}'.format(response.status_code))
 print('API Response:\n\n' + response.text)

There were a few errors, but they seemed to all just be local Bitbucket users. No big deal.

Problem #3: How do we update all of our Bitbucket projects with this new group?

Unfortunately, Atlassian does not provide a way to bulk edit project or repository configurations. This is terrible oversight in my opinion, but is out of scope for this post. Still, though…

To solve this problem, we turned to Script Runner for Bitbucket. We performed the work in this order to prevent anyone from running into errors.

  1. Add the new group to all of the projects.
  2. Add the new group to global permissions.
  3. Remove the old group from global permissions.
  4. Remove the old group from all of the projects.

We used the provided code snippets from Adaptavist. See docs here.

 

 

Minor Updates

  1. Working on cleaning up KillAllBindings a little. I’ve made some (probably) final changes that I need to test out. Once that’s done, I’m not certain what else I’ll be doing to/with it unless a bug comes up or there’s a need for functionality to be extended.
  2. I’m continuing progress to extending the testing quality of development builds internally while trying to address build speed concerns. Recently I optimized our build process and got build time cut by about 30%. We’re looking at other ways to speed this up all while performing more testing during the build process (which may offset the optimizations, but I’m cool with that).
  3. I’ve started working on a tool that will make source code collection for backing up source easier. It’s called ARC (Archive Released Code) and more information will come as I get closer to completion.
 

Process Enhancements

For a while now, we’ve required code reviews for all changes before they could be merged into a team’s trunk fork. This helped catch some obvious stuff and helped increase code quality. This stuff seems obvious but when it’s not part of the culture, it seems like a big feat.

The next step was to require the code to build before it could be merged. We implemented this about a month ago and so far it’s been a pretty smooth transition. Bamboo picks up the branches (annoyingly, it will build a newly created branch even if there are no commits but that’s another matter), build all of the solutions, and report the status to Stash. If the build fails, the pull request can’t be merged. Progress.

Next, we need to implement unit tests. One team has gotten that working in their fork so it’s just a matter of time to get the changes merged.

This is all text book stuff and it’s been neglected for a while, but it’s good to see us begin to enhance our process. It means more work for me, but it also means that there are less firedrills because less issues will slip through the cracks and into customers hands. I came from QA and I know what it’s like to be in the room with a pissed off manager, pissed off support agent, and a pissed off customer and it feels like it’s all your fault because you missed something.

 

Branching Strategy

We’re spending a lot of time discussing our migration from TFS to Git at work. We need this transition to go smoothly. I need this transition to go smoothly. There are a lot of doubters. Yesterday, I was talking to one of our senior DBAs about this and he said “I don’t care, I’m just waiting for it to crumble.” After I told him that I was the one driving this transition, he just sorta wandered off without saying anything else.

This is such a dramatic departure for all of our .NET developers that are used to Visual Studio being a one-stop-shop and I’m worried about backlash when we start making them use another tool (SourceTree since we use Stash).

Anyways, I wanted to go over our current TFS branching strategy and our proposed branching strategy in Git. Or, at a minimum, try to start documenting it.

Main----o---\-----------------o--o
Dev Branches \----o----o----o/       There's a dev branch per team.

The current issue is that Main is used to integrate all of the teams code. As you might imagine, Main breaks. We don’t have a “golden” copy. Additionally, “branching” in TFS is more like a forking operation in Git. A branch is a unique copy of a particular project from some changeset. The major difference is that when you branch in TFS, you’re ONLY getting that one branch whereas in Git, you get the entire repository.

This is what we’re proposing in Git.

Master----o------------------
Integration\-----------------
|
| Fork of "Main" repo per team.
|
Master - - - - - - - - - - - Teams will not use this branch.
Integration\----------------
Feature Branch\----o---o---o-^ Pull Request to merge into team integration branch.

At the end of the sprint, the dev lead will merge their teams integration branch into the “Main” integration branch. All of the other dev teams will need to either rebase their branches when another team commits their code to the Main integration branch or deal with resolving countless merge conflicts.

So, why not branches for these development teams? Well, we discussed this, but the assumption was that we could help keep everyone’s experience as clean as possible so they aren’t sifting through hundreds of feature branches.

What about releases? Hot fixing?

Release-X.y                   /------- We plan to branch and tag this. Potentially branch off of this for HFs??
Master----o------------------o
Integration\-----------------^
|
| Fork of "Main" repo per team.
|
Master - - - - - - - - - - - 
Integration\-----------------o
Feature Branch\----o---o---o-^

Enter fork syncing. Any branch made on the repository that is forked off of will get this branch. In the event of a HF, they will make their fix in their Release-X.y branch and then submit a pull request to have this code merged into the Main release-X.y branch.

Why the hell is release management not merging all this code and resolving conflicts? We’ll, to be frank, there are a couple reasons.

  1. We don’t know the code base.
  2. The assumption that any developer should be able to resolve a conflict simply isn’t true.
  3. The code base is massive and touches so many technologies that no one person can know how it all works (hence #1)

The only time release management will merge code is when we merge into the Master branch or Merge code into a release branch for HFing.

The big benefit to moving from TFS to Git is the ability to enforce code reviews by taking advantage of pull requests. Additionally, we have other development teams that work on mobile apps and Java apps that are already using Git (previously GitHub, now Stash) so it’s be nice to get everyone using the same technology.

Finally, this branching strategy is still in it’s infancy and we’re still toying with it to make sure that it breaks the KISS rule and to ensure that it’s sustainable for everyone. Also, forgive me if I’m using any terms incorrectly, there’s a lot of TFS lingo beat into my head.

 

Removing TFS Source Control Bindings from Solution Files

Today, I started looking into removing the TFS source control bindings from all of our solution files. At the time, I didn’t know how many solution files we had, but I knew it was more than I wanted to go through manually. I did some research online and even found some tools, but they didn’t work correctly for me so I wrote a Python script that would do this for me.

Basically, you edit a single variable, providedPath and run the script. It goes through a directory (and all of it’s subdirectories) looking for solution files. It then goes through each file, finding where they start and end (if they are present), re-writes the file without that section, and replaces the original file. I’ve ran it today, build solutions to make sure I didn’t break anything, and checked that code into Stash. I’ve also build the whole app successfully.

The script is located on GitHub here.

One thing this script does not do that I might add is deleting .vccss files. I didn’t bother with it since I’ve added that file extension to the .gitignore files.

This might grow to do additional things as we test migrating from TFS to Git. My guess is that this will not be the last time I copy all of our source over. Oh, and we have nearly 200 solution files, half of which contain source control binding sections. I’m glad I scripted that.

 

EDIT: I’ve taken the script off of GitHub while I overhaul it and determine if I can actually share it.