January 3, 2021
Building McPiper v1
During the first Covid lockdown, I had extra free time and decided to build McPiper - simple CI/CD monitoring tool that lives in the macOS status bar. Despite being a small application, it took me more than two months to build it.
When starting on this journey, I didn’t anticipate SwiftUI being so awesome but having a very steep learning curve at the same time.
Luckily, I kept a dev log and wrote down useful code snippets, links, problems, and solutions as I went along. This blog post is the summary of building the initial version of McPiper.
By the way, McPiper 2.0 is in progress, with support for on-prem GitLab servers, GitHub Actions, Circle CI, with the foundation to add more CI/CD tools in the future.
Subscribe to my newsletter to find out when McPiper 2.0 is out.
Scratching my itch
I hate GitLab pipeline email notifications. And I also hate when my unit tests and linting fail during the merge requests.
There are alternatives to the emails; you can use Slack or Telegram bots, but keeping online chats open during in-depth work sessions can harm productivity during these sessions.
The first Covid lockdown in the UK was quite strict, everything was shut down, so I had more free time. There was no commute, no gyms, no coffee shops, nor anything else open aside from grocery shops.
After the initial challenges of finding the toilet paper, yeast, and eggs in the shops, I started to get bored. Since I build apps full time, I decided to learn a new programming language and build a macOS app that would monitor the CI/CD pipelines from GitLab and let me know when it failed without the need to keep emails and slack open.
I had some experience with Swift a few years before then, but it was limited, and I inherited a codebase from someone else.
Starting with a blank canvas is a very different experience.
For web applications, you have generators (CookieCutter, Yeoman) that can create a starter for your project. The same exists for iOS apps, but it’s a different story with macOS apps.
macOS templates were mainly in Objective-C, and the setup includes too many features, which are great for full-blown applications, but overkill for a small status bar application.
For me, Swift was a better option since I had some experience with it before, and the syntax looks very similar to Kotlin and JavaScript.
The other option I ruled out straight away was Electron. The builds are about 40MBs, as you need to ship the whole browser. Rendering a few lines and few modal windows require less than 40MBs.
SwiftUI
While searching for Swift app generators, I came across a video on SwiftUI, which was released in 2019. It looked amazing, and it quickly reminded me of React; the concept is very similar. UI declared in the code can be reused, and for the data bindings, Apple is promoting reactive data structures, i.e., your views can subscribe to objects or fields in those objects, and your views will be updated straight away.
Previews in SwiftUI are great, too - you can quickly render the single component and see how it looks. Sometimes, it’s a bit painful to integrate previews with the data, but it’s due to my codebase code and data structures.
Since SwiftUI was released just a year before, it was still evolving - it was hard to find decent tutorials on it, and every problem was either focusing on iOS or referencing an old format or code that was already deprecated unusable.
If you run into an unknown bug, and when you start learning a new framework, you run into these a lot.
Googling these errors can lead you to the answers, but then you realize the format was an update or the functionality was already deprecated. It felt like SwiftUI was a beta offering.
A few websites were extremely useful. There are three; specifically, I came back to almost every day trying to wrap my head around how to use Combine and SwiftUI, how to pass data between object and the views, and other non SwiftUI related bits in Swift; these are:
Also, Apple Development documentation is awful, sometimes non existing at all. This was very frustrating. Most large open source applications and frameworks have docs that are light years ahead of Apple’s.
I read somewhere that it takes Apple docs teams a lot of effort to write and sign off the documentation.
But header files don’t have this level of bureaucracy, so many useful comments are added straight in the header files, which are available in the Xcode.
Starting Small
After some small experiments and the first successful compile, I felt very optimistic. I thought - let’s add as many platforms as possible (GitLab, Github, Circle CI, Travis, Bitbucket, etc.), so it quickly got too complicated even before the first working prototype.
Since the unknowns of Swift and SwiftUI added a lot of uncertainty, I decided to focus just on Gitlab as I use it daily. Otherwise, the application would be too large, too complicated, and I’m pretty sure it’d be full of tech debt.
With just one platform to support, I don’t worry about it too much. The codebase is small, and rewriting it from 0 wouldn’t be too daunting.
GitLab API
GitLab released GraphQL a few years ago, but it’s not well documented. For example, it doesn’t specify that you need full API access, not only read-only but write too, even if you intend to read-only.
My initial idea was to run pipeline checks every 1 minute, but some beta testers wanted to have checks almost every 5-10 seconds.
It’s ok if you have just a handful of repos to monitor, but when you have more than 30, then you need to make multiple requests in parallel.
I managed to find a way to retrieve data on ten projects in a single GraphQL query, so it saves many extra requests.
GraphQL usually calculates the complexity of the requests, so in my case, you can only get data for ten repos at once.
If you monitor 30 repos - 3 requests, plus not all data is available via GraphQL. So you need to make more requests to the REST API.
Despite GitLab claiming they want GraphQL to be the primary API interface, there were many inconsistencies with their current REST API.
For example, the REST API pipelines have the numeric ID, but GraphQL is actually a string with some extra path, which is an internal GitLab ID.
Luckily, GitLab has a high number of requests for auth users; it’s 2000 per minute, which is a lot. GitHub has 5000 per hour for auth users.
Status bar icons
It took me some time to understand how to change the icons programmatically. For icons and images in macOS, you can use multiple formats - SVG, png, PDF, but it was slightly more complicated for the status bar icons.
I’ve looked at some open-source repos to understand how they do it.
Telegram generates the image on the fly as they display the number of notifications you have.
In the end, it required some extra googling and trial and error to make it right, but the icon can be highlighted in red in case your pipeline fails, and it’s only swapping one image to another.
To correctly render the status icon, it’s best to use a PDF template with the correct size; these icons don’t autoscale. Plus, you need different versions depending on the dark/light views.
It turns out it’s easy to change it but takes too much reading to figure it out correctly.
In the end, it was just two PDF files, and you can change it in the code; you just need to store a reference when the app is initialized.
Combine
I was still trying to wrap my head around this and forgot most of the concepts behind it as I haven’t touched this part in a few months.
The second largest part of building McPiper 1.0 went into trying to figure out how Combine works.
This was useful as you can make some requests in parallel, combine the results, and process them in the same thread.
But it took me a very long time, very, very long time. With a lot of trial and error, plus reading into the resources on this, more trial and error, but then it made sense.
If you plan to learn Combine, make sure you check SwiftUI Notes by Joseph Heck. It walks through all Combine concepts and is very easy to understand.
Also, I wanted to use Combine with Alamofire, but the support for Combine was added around the same time I’ve started working on the app, so again, not much was available on the web.
Initially, I couldn’t wrap my header around sink vs. other Combine methods since I might have N number of requests.
Most examples for the Combine framework only have Merge 2 or 3 producers. But, there is Publishers.MergeMany which does precisely that.
And it took me some time to realize that .sink is an integral part for Combine. Otherwise, the results weren’t completed in the timeline manner; it’s also quite hard to debug as simple log statement wasn’t helping as they were executing at different times.
In the end, it turned out to be about 40 lines of code - chunking it into multiple requests, running them in parallel, combining, and processing.
Achievement unlocked - basics of Combine framework.
Beta testers
I was using MakerLog to keep the log of what I was doing daily for McPiper. It’s a great community, plus it’s a great productivity hack; marking something as done as part of your long-term goal motivates you. It does help you to keep the momentum.
When the prototype was stable enough, I’ve started a thread asking if someone is willing to test it, and I got the first few beta testers.
After the initial batch of testers joined, I posted in Slack #dev channel, and a few of my colleagues were interested in trying it too.
Microsoft AppCenter
AppCenter is basically a CI/CD for building, testing, and releasing different app types - they have support for iOS, Android, Windows, and macOS.
Unfortunately, the macOS builds produced in their CI/CD aren’t notarized, which is terrible if you plan to distribute the app and ask people to install it.
Lack of notarization displays the message - this app can’t be trusted, and you need to go into Security settings to actually allow the app to run.
The reason notarization is problematic is that it’s an async process. You submit a notarization request, but the response can arrive back only a few hours later.
This doesn’t work if you only have limited time in your CI/CD, plus there are no webhooks for getting a successful response from Apple when the app has been notarized; they only send an email.
After learning all of this, I’ve changed the tactic; the app was built locally, notarized, and then I used AppCenter to distribute the build to the beta testers.
AppCenter also has some great features like crash and usage (simple analytics) reporting.
Feedback and roadmap management
I’ve used Trello for roadmap management and Google Forms for feedback management.
Since I wasn’t sure if feedback or roadmap links would stay the same - they are just redirects on the website. i.e. (https://www.mcpiper.app/roadmap/)[https://www.mcpiper.app/roadmap/] and they’d redirect to the required page - internal or external.
The app that’s shipped to the end-users, every time you change this feedback URL, you’d need to update the app, and old apps won’t point to the right endpoint.
Managing the URL redirect on the website makes it easier to manage.
AppCenter has crash reporting; surprisingly, the app didn’t have any crash reports during beta tests. But the Apple review team managed to crash the app at the first launch of the app.
Website
I wanted to try GatsbyJS for some time - it’s total overkill for this use case, but it was interesting to try it out, and a small side project was a perfect opportunity.
It’s a static website generated in CI/CD and pushed to AWS S3 behind CloudFront. No databases or any moving parts.
App Store submission
When I submitted iOS apps for a review a few years ago, it took at least a week to get your app reviewed.
But I was pleasantly surprised when my initial app submission got reviewed in less than 48 hours.
And of course, it returned with a crash report, so I had to find a way to debug it and also replicate it.
If you ever need to analyze the crash reports and symbolize them, this is a very nice guide which helped me to resolve the issue - https://gist.github.com/bmatcuk/c55a0dd4f8775a3a2c5a and official Xcode documentation
Notifications
The initial version didn’t have notifications support.
And it was the first thing the beta testers requested.
The initial version was just updating the color in the status bar, which you won’t notice if you use another app in the full screen. If the pipeline is ok, the icon isn’t highlighted, and when the pipeline has failed, the icon turns red.
With adding notifications, the usability of the app increased dramatically.
It’s good to get external feedback. When you look at one thing from the same angle for too long, you start to forget there are other ways to look at it.
In version 1.2, I’ve improved notifications further. Added an action to the notification so you can go to the GitLab pipeline page straight by clicking the notification View action.
There are some limitations to how notifications work - at the moment, it just compares the status of the previous pipeline. It doesn’t take into account that the pipeline can be on a different branch or merge request.
Result
I’ve achieved the result I wanted - launched the app, learned a new language and framework. And the app is less than 8MB in size, there are two libraries that are actually optional, but they save me a bit of time and code. It’s possible to drive the size down further, but so far, it’s ok.
But it just took way too long.
With full-time work and less physical activity as it was a lockdown - I’ve burned out, so I took the next few months to recover.
I got too excited when I started working on McPiper; some days, I remember waking up at 5-6 am to work on this application.
Next steps
Currently, I’m working on the internals of the McPiper, so it allows for Github Actions and other systems to be added in the future.
Plus, I plan to integrate logs in the app, so you can view the logs without opening your browser.
Subscribe to my newsletter and I’ll let you know when I ship these changes.
And check out McPiper app.