Current Situation
The Research Software Engineering (RSE) department currently services around 70 research projects at any given time with the collective pipeline of projects just short of 100. Our department of 48 engineers are dedicated to supporting research and applying their diverse set of skills and experience to build research software for all manner of projects. However, managing the workload, relationships with researchers and ensuring a successful, collaborative project delivery can be challenging. Ensuring our RSEs in their capacity as developers as well as our senior RSEs in their capacity as project and service managers are prepared, supported and able to manage these projects flexibly but consistently is the first of our priorities in 2024.
Agile Methods and Scrum
Towards the backend of 2023, the RSE department started to experiment with using a lightweight implementation of an agile methodology which encapsulated the essence of agile values without being overly prescriptive. One popular implementation of the principles of agile project management and software development is “scrum”. According to the Scrum Guide, Scrum is a lightweight agile framework that helps people, teams and organisations generate value through adaptive solutions for complex problems. This, to us, sounds an awful lot like research!
Agile ways of working encourage a mindset where changes in requirements are the norm, and where the solution development process embraces that change. In Scrum, this is achieved by updating the requirements continuously throughout the duration of the project, and engineers work in short-term (2-4 week) cycles called “sprints” to deliver the next increment of the required output.
Although designed largely for software engineering projects, we are adapting scrum into a flexible, lightweight version designed to help us deliver our service effectively and consistently to all research projects. Naturally, it aims to be sympathetic to the high variability in output, research domains, size and required technologies associated with research at the University. Our scrum approach will be based on a consistent set of fundamental principles and processes but also be adaptable as necessary for each project to ensure we deliver solutions that meet each project's needs and are delivered on time.
Our Vision
Over the next 6 months, our senior RSEs with project and service management responsibilities will be undertaking “Scrum Master” training with University training provider QA. This course leads to a “Scrum Alliance Scrum Master” exam and resulting certification. Alongside this, Ann Gledson, one of our senior RSEs, has been developing a two-day workshop called “Agile Methods for RSEs” which formalises our lightweight scrum approach. This workshop will be then run for the whole RSE department to ensure all RSEs are trained, with practical experience, in the new approach.
Over this period, our academic customers will start to see the introduction of new tools and techniques in the day-to-day running of our projects and the behaviour of our engineers. We believe that these changes will ensure a better, more consistent service and appreciate your engagement. The use of Kanban and issue tracking will become commonplace, for example, and GitHub will form the nucleus of project activity. However, if you already have alternative agile tooling in use, we will continue to remain as flexible as ever and look to use your tools if it best serves your needs.
Evaluation Course
We recently ran a one-day preliminary version of the course for a volunteer group of RSEs from the department. We were very pleased with the feedback we received, both positive and constructive, and are confident we will deliver a course that is informative, useful and hopefully fun too when we roll it out in the next few months!
The day began by teaching the theory of agile methodology foundations and the Scrum implementation, why it is useful for our department, and how it can be adapted carefully for different types of projects. The majority of the day was then spent doing hands-on training, applying the process according to our course handbook through a practical series of exercises.
The eight RSEs split into two teams, each with a “Scrum Master” and three “Developers”. A toy project was introduced: to develop some frontend software for presenting and querying data on a map. Requirements were elicited from a scope invented in advance and recorded using our standard RSE Request Document template. Each team created a GitHub project, including backlog and Kanban board, as well as a code repository from our new set of GitHub templates.
Two sequential sprints were then performed by each team, each lasting a couple of hours and included all of the key Scrum “ceremonies and artefacts”, including the creation of a “Product Backlog” (a list of all project requirements broken down into tasks), a “Sprint Planning Meeting” to create a “Sprint Backlog” (all tasks to be completed in the current sprint), an hour of development work and finally a combined “Sprint Review” and “Sprint Retrospective” meeting. RSEs took it in turns being “Product Owners” (a key person who is responsible for what work gets done and represents views of all project stakeholders). As “Product Owners” they put themselves in the shoes of their customers and prioritised the issues using “MuSCoW” categorisation (i.e. labelling each end user requirement as being “Must have”, “Should have”, “Could have”, “Won’t have”).
Here are some examples of what our RSEs thought:
Andrew Rowley, Scrum Master – “I have worked on a number of open-source projects in the past, working in an agile way, but had not used the Scrum methodology directly before. It was very interesting to see how this can help to manage the process and get people involved in the projects; in particular, allowing the person you are developing the software for to add and change their requirements directly in the project [Kanban] boards, allowing them to see the whole process as it unfolds, but also giving the power to the development team to work on those requirements in a way that they can use their expertise to get the tasks done. The task itself was easy enough to get to grips with but also didn’t get in the way of getting to grips with Scrum itself which is what the course is really about.”
Emma Simpson, Scrum Master – “I really enjoyed this training, it gave me a chance to put my recent learning from the scrum master course into the context of our RSE team and share my own knowledge of how scrum works in practice with others. For me the main value of this course is how it facilitates knowledge transfer from different RSEs on how their projects have worked and how scrum and agile techniques could make things easier and more efficient using direct, lived examples.”
Scott Archer-Nicholls, Developer – “I work on a relatively large project, and our team has been using Scrum since June last year. This course was very helpful to help solidify my understanding, as well as to gain insight into how others are using similar methods, particularly to see how they can be applied flexibly to projects of different scales. It was a very engaging workshop with lots of fruitful discussions, and the mini-project was a really fun group exercise to apply what we’d learnt!”
Adina Rahim, Developer – “"I've been using Scrum methods since last year, and while they've proven helpful, grappling with unclear user requirements presented some challenges that the agile training course effectively addressed. This training equipped me with invaluable techniques to clarify requirements, break them down into actionable tasks and subtasks, and prioritize them using MoSCoW labels. I believe it will provide developers with essential skills to navigate ambiguity, set realistic expectations, and deliver high-quality products that exceed stakeholders' expectations, ultimately enhancing our ability to achieve success."
The Future
Our aim in the longer term is to run this workshop once per year as both a refresher for existing RSEs as well as to ensure new starters are trained to the same standard as our existing engineers. We also shared some of our material via the Manchester Research Software Community. We would also like to open this workshop up to researchers and RSEs outside the RSE department if there is an appetite for us to do so. This is just the beginning for our Agile Methods for RSEs course, so watch this space, and do get in touch if you would like to know more about it.
Feedback
If you have any questions or concerns about the on-going changes in the RSE department then please contact the Head of Research Software Engineering directly – we are happy to listen. If you would like to find out more about the course and let us know if it could be useful to your research group, contact Ann Gledson.