Breaking things one pointer at a time

Automatically updating git submodules using GitHub Actions

I know that the mono-repository is in style at the moment, but git submodules are a fantastic (and probably over complicated) tool for being able to store components of a git repository that may need to be kept used, but separate. For example, you may need to track a specific upstream version of another repository that isn't controlled by you or your organization for use in your repository, or you might want to mix repository visibility or share some code between different repositories that need to be separated.

I personally use submodules in the repositories for a wide variety of uses. For example, my research project uses a submodule to track the latest version of the codebase that I am modifying so I can write patches to match against those versions. Another example is my site, which has a bunch of submodules to track various things which get published there, from my resume and my research project, to the theme that I use for both my site and this blog.

I only recently moved to using the same codebase for the theme of my site and blog, which has both made my life easier and caused me a great deal of heartache - and almost all of the heartache came from git submodules. You see, submodules are tied to a specific commit, and running a command like git submodule update (which you'd think updates a submodule to the latest version) only checks out the submodule to the same commit as what your local repository already has stored. This makes life a little more difficult - how can I easily update submodules without having to specifically know the remotes, branches or location of all of my submodules?

First, all of the submodules have their remotes and locations (and optionally branches) stored in the .gitmodules file, which can be used to iterate through the submodules and pull down the latest versions. Another option is using the git submodule foreach command to update the submodules by fetching the remotes and then checking out latest commit, which has the disadvantage submodules not (by default) being checked out branches, and instead is checked out to commits. Both of these options are obviously not particularly ergonomic to work with, and so git introduced the --remote flag to git submodule update to tell it to update to the branch tracked on the remote instead of the local repository.

Great, so now we know update our submodules to the latest remote commit (huzzah!), but I want to push to one repository and watch the other repositories automatically update to track that commit (instead of requiring me to run a command). How can we do that?

GitHub Actions to the rescue! (although unfortunately it won't quite get us all the way there)

GitHub Actions allows you to trigger commands on arbitrary events using the repository_dispatch event. The first step is to create a GitHub Action in our repository that updates submodules for us.

Of course, the simplest way to do this would simply be to recursively clone the repository using GitHub's checkout action, to run git submodule update --remote and then git push the changes back into the repository. Unfortunately, this doesn't quite work out for two reasons; we won't have write access to the repository and we won't be able to clone private repositories. We can get around this by passing a personal access token to GitHub's checkout action and telling it that we need to pull submodules recursively. An example of this action can be found below, and is actually what I use in my site's repo.

name: Update module
    types: update
    runs-on: ubuntu-latest
      - uses: actions/checkout@v2
          token: ${{ secrets.PAT }}
          submodules: recursive
      - name: Update module
        run: |
          git submodule update --init --recursive --checkout -f --remote -- "${{github.event.client_payload.module}}"
          git config --global "GitHub Action"
          git config --global ""
          git commit -am "deploy: ${{github.event.client_payload.module}} - ${{github.event.client_payload.sha}}"
          git push

The second step is to create the trigger for the repository_dispatch in the submodule repositories themselves. This can be done using the excellent repository-dispatch action to send the event. An example of this can be shown below.

name: Dispatch to repo
on: [push, workflow_dispatch]
    runs-on: ubuntu-latest
        repo: ["owner/repo"]
      - name: Push to repo
        uses: peter-evans/repository-dispatch@v1
          token: ${{ secrets.PAT }}
          repository: ${{ matrix.repo }}
          event-type: update
          client-payload: '{"ref": "${{ github.ref }}", "sha": "${{ github.sha }}", "module": "owner/submodule", "branch": "master"}'

This of course works with submodules that you control, but what about for submodules that you don't control? Unfortunately there's no way to start an action based on a push in a different repository, nor is there a way to create a webhook that you could use to trigger another action from a repository that you don't control. As such you are left with two options; to simply run the update action at a frequent interval or to hack together a way to do it with activity email notification and a repository_dispatch. I know which of the two I'd rather implement in a hurry!


On the enjoyment of work

I'm of the opinion that in society, the general attitude towards jobs is that you should "do something that you're passionate about". If medicine takes your fancy, then take up your stethoscope and get to it. Does the thought of court put you in a tizzy? Find a briefcase and start practicing law. Is graphic design your passion (yes)? Then grab your wacom tablet and get ready to draw.

This is an attitude that gets taught to kids and is especially pushed throughout high school as teenagers get closer to the end of the secondary education and begin thinking about their adult life. I remember being told to "picture what I want to be doing in thirty years" and to start taking steps towards doing that when I leave high school. For me, I this was not exactly great advice - I had no clue what I wanted to pick between music or computer science! Whilst I eventually decided on doing computer science, I definitely understand the reason why there's so many people doing music, arts or history degrees - its what they've been told to do! And if there was a high demand for professions in those areas, they would be able to spend their lives doing a job that they loved. Unfortunately, given the number of people taking the courses and the general lack of capital-creating potential, its unrealistic that all of them would be able to get gainful employment, especially as the expected age of retirement seems to increase. I don't think that this is necessarily a point against doing the arts as a degree (again, I almost picked music, and my brother is actually studying French Horn), but I think that the wish to be entirely fulfilled by your full-time employment is both dangerous and misinformed.

I've heard of engineering processes being like a four-sided piston several times - you need to have a balancing act between quality, cost, time and scope, and as each of them is moved, the others also need to be readjusted. I think life is much the same way, there's things that need balancing and we can't have them all going at full steam. I can't be all in on my work and spending massive amounts of overtime whilst also trying to manage a family, stay in touch with older friends and keep a hobby. It's entirely unrealistic.

Most workplaces recognise this (or are forced to in Australia by the government) and have a maximum set of hours that you are paid to work - everything else is overtime. So long as you aren't taking an enormous amount of time to commute to work and aren't continuing to work past your hours, that leaves roughly a third (or a quarter when factoring in eating) of the weekday - plus the weekend - to spend on things which aren't work - family, friends and hobbies. All of those are great things with which to be fulfilled.

My maternal Grandfather spent working years as a clinical biochemist. He has long since retired and his "hobby" is now his employment; gardening. He's become so proficient at caring for plants and identifying native flora that he's become an invaluable member of several parks in helping with botanical management. Whilst I know that he enjoyed his time as a clinical biochemist, it was not his "be all and end all" to the point where it was where he gained his fulfilment from, instead he got fulfilment from his family, friends and hobbies.

This is much of the same approach that I've taken to music. My gainful employment will be coming from software development, something that, yes, I enjoy, but that doesn't mean that I'll stop playing music. Over the course of my time at university, I've continued my involvement in music, and by my own estimation am a significantly better musician now than I was then. Whilst I am certain that the total improvement is likely not as great as if I had formally studied music, having music as a hobby that I enjoy and can gain fulfilment from without relying on it for my finances is both freeing and adds to my enjoyment of it.

I think that this is the right balance of things. Even in jobs in which you would be theoretically fulfilled, you can still have things go wrong. Sudden deadlines, a difficult workmate or an overbearing boss can all make jobs significantly less enjoyable. If your previously fulfilling work is no longer enjoyable, then what are you enjoying?

Thus my charge to you is this; don't put all your eggs in one basket. In the end, work is work and should be approached as such. It is a means to an end, to give you an opportunity to live the other parts of your life and shouldn't be what your entire life is about. So take up a hobby, spend more time with your family and friends, or start a job that you know you won't enjoy but can use to further one of them. I'd bet it'd make life more rewarding.

(disclaimer: author is an university student who hasn't ever had to apply this advice in practice and is writing at about 1AM. Take it with ten grains of salt)


Imposter syndrome and improvement

I've recently been watching a lot of Daniel Kapadia's (a.k.a. ddk) videos. For those that don't know, ddk is an ex-Quake professional, who's moved into esports casting with a primary focus on FPS games. In a (somewhat) recent video of his, he says the following: limiting factor I had was - and I still suffer from this a great deal - and its part of the reason why, in some respects, I've achieved anything in some of the realms that I've worked in is impostor symdrome. It's been an issue for me.

It's the thing that really makes me work really hard sometimes, but it also is the thing that makes me discount myself from feeling like I'm worth anything or good at what I do in any remote sense. So that can be a really big issue for me, and it was an issue when I was a player...

This phrase reminded me of a rule from the "Digital Age" addition of Dale Carnegie's "How to Win Friends and Influence People"; Give Others a Fine Reputation to Live Up To. In many senses, I think that impostor syndrome is the other side of the coin for "giving others a fine reputation to live up to".

The definition of impostor syndrome is to doubt your accomplishments or talents and to fear being exposed as a fraud. If you are given a "fine reputation to live up to", especially if its not for something that you've actually done yet (as "How to Win Friends and Influence People in the Digital Age" suggests), then of course you'll have impostor syndrome. You're being expected to live up to some ideal which you haven't achieved yet, and whilst being treated as though you've already achieved that ideal may be nice, you are - in the literal sense - an impostor.

"How to Win Friends and Influence People in the Digital Age" suggests the following:

Coaches, mentors, leaders, and parents often find that people live up to our expectations of them, no matter how diminished those expectations are. If a man feels unimportant or disrespected, he will have little motivation for improving himself. So why not create a vision of him that embodies everything you know he is capable of achieving, as well as everything you don't know about his possibilities? You will rarely be disappointed...

To change somebody's behaviour, change the level of respect she receives by giving her a fine reputation to live up to. Act as though the trait you are trying to influence is already one of the person's outstanding characteristics.

If this isn't a call to the bettering influence of impostor syndrome, I don't know what is!

We must also keep in mind the other component of the quote I've taken from ddk. Impostor syndrome can cause people to "discount themselves from feeling like they're worth anything or good at what they do in any remote sense." - in other words, it can cause low self-esteem. So there's certainly some skill to the balancing act of trying to make people live up to the expectation of their better self without massively impacting self-esteem.


Honey, I shrunk the blog!

At the time of writing this post, loading the front page of this blog without anything cached takes around 1MB of data. Compared to the average website page (according to the http archive), this is actually pretty good. The median page apparently ranks in at around 2MB of data, with about 1MB of that being images, and another half a megabyte of JavaScript. Even worse, according to the http archive's 2019 page weight reports, 10% of website pages are over 6MB, resulting in requiring at least a second to load the page on many internet connections around the world.

Obviously this blog is no where near as bad, so why have I taken this entry to talk about page size and how I've made this blog smaller? (Yes that's the topic if you haven't picked it up already.) Well this blog is almost entirely text, with one image that is used for the header/footer, and some custom fonts. There is no JavaScript, no videos, some CSS (although not that much locally, only 9KB is CSS that is hosted here), so why on earth is this website checking in at even half of what the "normal" website would be?

Well the answer, as it turns out, is entirely that one image that is used for the header and footer of the website. The profile image (which I use as my profile picture in most places) is a 2320x2314 JPEG image that takes up 747.8 KB of space. That's an absolutely whopping three quarters of my entire page, for one image that arguably isn't even needed. Fortunately, there's ways around this.

The container that my image sits in at the top of my page is a square with a side length of 125 pixels, so this image only actually has to be 125x125 for there to be no noticable change in quality from the original. The container at the bottom of the page is also a square with a side length of 30 pixel, and as 30 is not a divisor of 125 it makes sense to create an image for that separately so that the image doesn't sub-sample oddly.

The static site generator that I use is called Zola, and has a number of inbuilt functions in its templating. One such function is the (very handy) resize_image function, which (as you'd expect) resizes an image to be of some particular dimensions. The function processes an image for you and returns its path, allowing you to keep almost everything unchanged save the loaded image url.

-      <a href="{{config.base_url}}" style="background-image: url({{config.base_url}}/img/{{config.extra.profile}}"></a>
+      <a href="{{config.base_url}}" style="background-image: url({{resize_image(path='../static/img/' ~ config.extra.profile, height=125, width=125, op='fill')}}"></a>

By applying this function, I ended up with images of sizes 4.15KB and 1.21KB for the 125x125 and 30x30 images respectively, a total reduction of 742.4KB. That's basically a saving of the size of the entire original image, with no discernible change in image quality!

Of course, after this change I was greedy for more gains, and thus began my fight for fonts.

I use a number of fonts on this blog (and on my main website), which are generally downloaded from Google Fonts on demand via CSS. Raleway is used for the text of front matter (the information for each blog post such as the publishing date, total words and tags), whilst the excellent Montserrat is used for the rest of the text elsewhere. Whilst I have replaced my Raleway with a combination of Verdana and Geneva, Montserrat will be sticking around as (in my opinion) one of the best looking variable weight sans-serif fonts available. Thus, my savings with actual fonts was only around 20KB, relatively paltry gains.

 html, body {
     background-color: $background;
-    font-family: 'Montserrat', 'Raleway', sans-serif;
+    font-family: 'Montserrat', 'Verdana', 'Geneva', sans-serif;
     font-weight: 400;
 .frontmatter {
     color: $title;
     font-style: bold;
     list-style-type: none;
     font-size: 14px;
-    font-weight: 700;
-    font-family: 'Raleway', sans-serif;
+    font-weight: 500;
+    font-family: 'Verdana', 'Geneva', sans-serif;

My major gains in the fonts department actually came from another place entirely... FontAwesome. FontAwesome is a fantastic font for using company logos instead of images, which does indeed save a lot of data compared to the size that even one image of a logo would be. The only problem is that the entire stack of FontAwesome icons that I'd need to get for my little symbols next to my website, GitHub and Atom feeds is around 130KB, which (now that we've shrunk a large portion of the website) accounts for about half of the remaining website. Fortunately for me, FontAwesome provides SVGs of their icons, which allows me to download them all individually and use them instead in <img> tags. The total size for the three icons individually is a little under 5KB, around 25 times smaller than the entire font.

-  <li><a href="https://{{}}" target="_blank">website <i class="fab fa-firefox" ></i></a></li>
+  <li><a href="https://{{}}" target="_blank">website <img class="symbol" src="/symbols/firefox-browser-brands.svg"></img></a></li>

According to Firefox's network tools, I'm now down to 128.2KB, of which my remaining font of Montserrat makes up a solid 70%, so there's clearly still some room for improvement. Perhaps I'll continue to look for an alternative font to use instead.


The curious case of the disappearing ideas

I've taken to setting aside time in evenings to write these entries (and very late in the evenings - its currently 11:40 PM). This has mostly been fine in terms of my schedule. I have university during the day, and I also often have other activities such as choir in the early evenings, so the late evening is pretty much the only time that I can write my entries without having it potentially disturbing the rest of my day.

But this has also lead to another thing which has arguably already negatively impacted this blog - I've forgotten almost every idea of what to write by the time I reach my allotted writing time.

Sometimes, I have a good idea in the middle of the day, or something to discuss that I've just done, and either of these would be a good idea to type up as an entry. But by the time its the evening, after the vast majority of the day has gone and my focus begins to turn to the next day, I find it incredibly difficult to remember the details of my thoughts, the minutia of my activities or the ideas that came to me throughout the day. Admittedly, this isn't a particularly promising thing to admit at what is effectively the restarting of this blog, but in my opinion it is better that I am aware of this problem (especially as it would be likely to persist if never recognised) and am now able to think of methods to combat it.

One potential idea for me to do away with my structured day and instead to write whenever the ideas come to me. This is somewhat tempting, because starting to put down ideas when they are fresh often means that the ideas will all be left semi-formed for me to come back to later to fill in. I could quite easily see (however unlikely) my drafts folder reaching peak capacity, with too many half-formed (and likely duplicated) ideas left strewn around that I am left with the paralysis of too many choices. The main reason why I wouldn't necessarily want to do this approach, however, is because of the structure that I need for the rest of my life. I'm currently in the final semester of my Masters, and as I have previously mentioned, I am in the process of writing my thesis. Writing a thesis is a time consuming process, but I am also taking a full course-load in addition to my thesis, resulting in even more time being dedicated to university. Thus, I require a large amount of rigid structuring of my day in order to ensure that I am able to get all the work of the semester done (and its arguable that I still don't have enough structure to comfortably make it to the end of semester with full marks).

Another idea is to note down ideas in a journal of sorts, with none of the ideas as fleshed out as they would be (this is arguably just a variation on the above). I already keep a small Moleskine journal on me to keep track of my tasks for the day, which has often proved invaluable in ensuring I don't forget assessments. Without modifying my day much, I could quite easily take short notes of my day and my ideas for this blog. The disadvantage of this approach is simply that I would likely need to provide myself some context for each thought in order for my previous thoughts for each topic to be able to be recanted as I had originally thought. Even of the topics I have already covered, if I had just written the title in my journal, I doubt that I would have been able to restate all of the points that I have.

Even as I struggle with this, I am lead to be incredibly impressed with the various YouTubers that I see uploading new content on a regular basis. Whilst it is arguable that some topics practically write themselves (product reviews or political news come to mind), the variety work between those topics needs to be interesting for their existing audience to continue to want to watch it. I think an example of this is LinusTechTips, who upload six videos a week on their main channel (and even more on their other channels). Whilst the majority of their content is product reviews or based on technology news, in the last two weeks alone they've included videos about a Chinese x86 chip, a budget build from the jankiest origins, run a "tech support" challenge, done a video on the custom air conditioning in Linus' home, discussed a pyramid-shaped case that was requested and looked at the remaining players in the (arguably dead) sector of mp3 players. That's an incredibly wide range of topics, all of which appeal to their audience and all of which have an incredibly large number of views, but all entirely original content ideas. That they have been able to keep up with the insane schedule of new videos, all whilst keeping fresh and original ideas coming over a matter of years is incredibly impressive to me.

I suppose I should hope that I can do the same.