Here at Carbon Co-op, our in house software engineering team has been working on our online presence for a little while now. We’ve been thinking about what digital services we provide now, and what new services we want to provide in the future – and we’ve been laying the groundwork for them. This is the first in a series of blog posts about that process.
We’ll start off by talking about data, which is the lifeblood of digital services. Digital services at their core are interfaces to data storage, retrieval and processing systems.
Spreadsheets, spreadsheets, everywhere
Carbon Co-op has a fair amount of data. And like in a lot of organisations, that data is stored in a variety of places. We have spreadsheets with data about members and people who’ve participated in projects. We have data stored in external systems – like our ticketing system where email enquiries and membership queries come in. And we have data stored in our own online services, such as our home energy assessment tool, My Home Energy Planner.
However, a lot of that data is not readily available, either to its members or within Carbon Co-op. That means that if someone wants the measurements of their house from their home energy assessment, for example, it’s not obvious how to get them. The data is stored, but it’s not available. Or sometimes, a staff member has someone’s phone number but no-one else does, which means it takes longer to resolve an issue than it otherwise would.
A key challenge for us, then, is in organising all the data we hold in a way that provides value, both to our members and internally. As we grow and take on more ambitious projects, it’s becoming more and more important that we get this right.
Enter the CRM
At the beginning of 2018, when we started grappling with these questions, we realised that what we needed was some CRM software. CRM stands for ‘customer relationship management’, and is a general name for software that helps organisations manage their interactions with their customers. In our case, that means our members, and the contractors that we talk to and train, among others.
There was a lot of data we wanted to stop keeping in spreadsheets, or on external websites, and start putting in a CRM. We wanted to keep information about, for example:
- membership status and history (as a community benefit society our 185+ members are our key beneficiaries and ultimate owners!)
- what equipment we had installed in members’ homes
- who attended what trainings
- what trainings contractors had attended, so we knew who we could recommend
We initially decided to use CiviCRM as our CRM software. CiviCRM is widely used in the third sector for managing donation drives, casework, and membership databases. It’s open source, so anyone can work on it, and it has an active community of users and developers. The idea was that CiviCRM would act as a central place to store all the data we had about members, and that we could act on it from there.
CiviCRM
The migration of our membership spreadsheet to CiviCRM went well. It was an web-based system rather than a file synced over a cloud storage platform such as Dropbox, which avoided editing conflicts. And we got features like renewals tracking and email templates.
Yet the functionality we wanted wasn’t all there. We wanted it to automatically record PayPal payments against memberships. But because the payment subscriptions were set up using a different system, there was no way to do that. And the membership functionality wasn’t very customisable. Things that seemed like they should be easy to automate weren’t.
What we found was that while Civi was adequate for managing membership, it generally wasn’t a great fit for our needs:
- It didn’t contain integrations for all the things we wanted. For example, we wanted to import information about event attendees from Eventbrite and Meetup, but this function didn’t exist. Civi has a built in events registration system but it’s not very user-friendly.
- It had a lot of features we didn’t need, the interface was unfriendly, and it was complex to deploy and keep up to date.
- It wasn’t easy to talk to from other web applications. When our developers tried to get data out of Civi, what we got back was in unhelpful formats and had little documentation.
We had a look around at other CRM systems but we didn’t see anything that would fit our needs. Most CRM software is built around identifying customers, marketing things to them, and tracking sales. That wasn’t what we needed out of a CRM, and Civi did seem like the best off-the-shelf software available for us.
To get what we wanted, it seemed like our only option was to create the functionality we wanted. We were worried about the cost of paying someone to make Civi into the software we wanted to be. There was a large upfront cost in paying contractors and good results couldn’t be guaranteed. And then there was an ongoing maintenance cost that we would be tied into.
But at the same time, building our in-house capacity to work on CiviCRM seemed like a bad deal too. It would take a long time given the underlying complexity of what is a specialist bit of software. And again, we would be locked into CiviCRM in the future and we weren’t sure that was a good idea either.
Our digital service ambitions
At the same time that we were evaluating CiviCRM, the Nobel Grid smart metering project was ending and we were planning future projects such as People Powered Retrofit and OpenDSR. It became clear that our future ambitions meant we would be creating new online services (or offline services facilitated by online tools). And we’d need to expand our team of software engineers to do that.
What we wanted was custom to our specification and innovative. We’d be building tools to help manage retrofits. And we needed an online service for members to take part in domestic demand-response trials.
Given this, we started re-evaluating our use of CiviCRM. It probably makes sense if you are an organisation that routinely does fundraising or casework – since that’s what it’s designed for. And if you don’t have in-house developers, it makes sense not to build something from scratch and tap into an existing ecosystem instead. But neither of these applied to us.
Next steps
Knowing that we’d have more developer capacity in future, it made sense to stop any further effort to make Civi into what we wanted. We decided to keep the membership system there for now, though, as we didn’t have a better option.
Instead, we would develop our new digital services using widely-used languages and tools. Python is a popular programming language, and Django is a high-quality and and widely-used web framework. This was a big plus over using CiviCRM. There are many thousands more people who can write Python applications than can write CiviCRM extensions. This means that we can easily recruit more staff to work on our projects, and at a reasonable cost.
Looking at the requirements we had, it was clear we would need to integrate lots of different services in one place: a hub. This hub would be the place where users could access all the data we hold about them. It would also be a portal to online services such as smart meter access or scheme participation.
Internally, Carbon Co-op staff would use it to see the data we hold about members in a single place. This would include contact details, membership status, support tickets recently opened, or recent home energy assessments. As we grow, this will help us provide high quality support to our membership.
And that’s how we ended up with the Carbon Co-op Hub.
Continuous delivery
Since these decisions were made at the end of 2018, it’s been full steam ahead. The first public piece of functionality on the Hub was the new member signup form. This lets new members sign up in a streamlined way, dealing with payment processing and creating new user accounts.
The second public piece of functionality has been our smart meter data viewer. This allows members with smart meters from the UK-wide rollout of smart metering to view their data online.
(NB: This doesn’t work for all smart meters yet, because our data supplier isn’t linked up to them all. But they’re working on it! We invite all our members with smart meters to try it out and see if it works for them at hub.carbon.coop. We’ll have another blog post about this soon from our new developer, Peter.)
We’re currently integrating the back-end elements necessary to support the role out of the People Powered Retrofit service in the autumn.
Behind the scenes, we’ve been busy too. We log PayPal and Direct Debit payments directly into our membership mangement software. And then we’ve had some time inbetween projects to build new membership mangement software into the Hub. It’s a lot more customised to our particular requirements than Civi was and requires less admin time. It paves the path for a more ambitous building out of our CRM software to handle contractors later this year.
We’re very happy with how this has gone and we’re looking forward to announcing a steady stream of functionality over the coming months and years. Expect more articles soon about particular things we’re building or have built, and the processes behind them.