16 Oct 2023
Every year on the first of October a band (which gets together with the sole purpose of making a record) meets in a studio and records an album in a day, or 12 hours to be precise.
It has become a bit of a ritual to listen to the album upon release, just out of mere curiosity.
The act of making music or art under constraints has always intrigued me, and music is even harder because you can so easily fall into a flow state where it feels somewhat… wrong… to leave something unfinished. Yet there is the flip-side of this which is that it is uniquely beautiful to hear something as it ended up, a bit bare, something that didn’t undergo the grueling (and often most time-consuming) act of tinkering with a song until perfect.
And the results are always raw, random, and beautiful as-is.
The latest album called “Across the Road” was filmed in Abbey Road Studios, and is absolutely amazing. From the process of creating the album to the final result, year upon year they have improved; it feels like the first years were about “can we do it? rush!” whereas the latest one feels more like “we know the drill, let’s put effort into where it belongs, and do revisions and add the non-essentials towards the end”, and it goes to show… the song structures, the arrangements, the little funny bits here and there, yet grave seriousness of some of the other tracks provides a very nice mixture of textures, themes, genres, and more.
The mere fact they recorded it in Abbey Road (using some of The Beatles’ instruments) was a source of inspiration that transpires through the musical themes of the album.
Hats off to Rob and Andrew as always, and looking forward to next years’ edition!
16 Oct 2023
I’ve been on call for about 10 years. I wanted to share a few thoughts on this, such as: Managing stress, how things change over time, and how the tech landscape and architecture change the scope of responsibilities of the person who is on call.
When I was asked to be on call nearly a decade ago, we had about 4-5 people who were selected as the group to be on call. This would mean we’d need to be available 1-2 days a week, in full mental capacity (aka not intoxicated) to jump on calls, debug issues, be woken up at any point during the night to help mitigate production issues. On the upside you get compensated, based on how many days you are on call per month, as well as how many hours of extra work you did as part of emergency responses.
At first thought it might seem a bit daunting, what if you randomly get a message to go out with friends? What if you don’t know how to fix the issue? What if it is outside of your domain and you need someone else to help? What if they are not available? What if you are not feeling well that night? What if you just pulled a full day and night of working and you finally manage to get some shut-eye just to be awoken in the middle of the night, knowing you have a fully packed schedule the next day?
Personally, I feel like the initial fear of being on-call outweighs the actual job; yes it will happen, you will eventually get the dreaded ping, but more often than not the issues are easier to fix (specially with modern tooling) than you might think; as long as you have a modern infrastructure with pipelines and auto-scaling of some sort.
What is expected of me?
When you are on call you are expected to know how to investigate, narrow down, debug, inspect network requests, try to isolate the issue to a specific service or area, then prod at it, check the pods, check the database, figure out if you can restart or redeploy previous pipelines.
You are not expected to be an expert in every domain, nor every part of the codebase. A critical mindset is what is needed, one that doesn’t go down a rabbit hole at 3AM but instead tries different high-level approaches to understand the problem from a few angles before trying to dive a bit deeper.
What happens when shit hits the fan? It is 2AM and you get paged. The initial response is a bit of fear, the alarms are going off, you have to get to the laptop and catch up with the chat messages. A few minutes in you have seen the ticket, caught up with the chat logs, and you are now checking the error tracking issue. Shortly afterwards, you work together with support to reproduce it, and meanwhile you figure out if it was related to the previous deploy. If it was, you start the rollback (after notifying everyone, in case there was something that the rollback would revert to, for example if the last deploy was a hotfix). If it wasn’t, you start the investigation, checking if flags changed in the database, checking access/error logs, checking pod restarts, etc. In some cases it will be out of your control and you’ll have to ping infra/devops, maybe they have an on-call system of their own, in some cases it won’t be backend but frontend and you might have to reach out to the on-callee there, or attempt to revert their own pipeline, but in most cases the issue is at least controllable.
The primary purpose is not fixing the underlying issue, it is getting the system under control so that normal operation can resume, so that users aren’t blocked and can use the application. Mitigation as a first line of response, fixing can take place afterwards, as long as the issue will definitely not resurface. It is an exercise of estimating the severity and likelihood of recurrance, and making the best decision with the information available at the moment.
How often will I be needed?
In practice, I don’t think I was paged more than 20 times in the years I’ve been on call, the majority of them being in the earlier years.
The size of the company and the stage that it is in often inversely correlates with the amount one gets paged. What I mean is, with time and growth, services will become more stable, certain areas of the codebase will change less, and more teams will become owners of their own domains; in some cases each team might have a separate cloud account hosting their own services, thus potentially moving the on-call responsibility away from a central group of people are more towards separate teams having their own schedules.
Depending on the size of the company and when this happens, it could either be a good thing or a bad thing, in some cases one person has been the sole person on call for month after month, due to the specificity of that domain and the lack of more people to share the responsibility with. I’d argue in those cases it might be worth it to bring it up to management, and see how to mitigate it, either by introducing more people to that codebase (people from other teams helping to distribute the responsibility of being on-call), or hiring more people / transferring internally.
When the company grows, you will likely adopt different pillars (either by building products or acquiring other companies to grow your portfolio), this means the knowledge of domains (and even access to accounts and resources) may start to split, only giving access to those who need it. This also means multiple on-call teams may start to spring up. Simultaneously, helpdesk support people must know which team (and in some cases which domain) to assign tickets to, which might trigger a page depending on the severity of the issue.
More importantly, by the time you’ve reached this size you’ll probably have a dedicated cloudops/devops team (or person?), and given the prevalence of “the cloud”, likely they will become the first line of defence when issues happen. Since we started containerising everything, multiple failures on a container or pod might trigger it to become unavailable, and new containers might spin up. This has the downside of potentially hiding issues, but the upside of it “fixing itself”. Thus, developers are not AS incentivised (as in the 2000s and 2010s) to produce error-proof code. This also means that the infra team might be notified first when a spike in container creation happens due to bad code; ultimately the fix might be to roll-back to a previous image, thus not even needing to wake up the backend engineers.
You might think I’m exaggerating, but eventually you will come across kubernetes.containers.restarts over environment:production,kube_namespace:project was > 20.0 on average during the last 5m and it will sink in. These are weird times indeed.
In any case, if you are considering being an on-call engineer and wanted to know more about what it requires, and how it will evolve over time, hopefully this gave you a few insights.
30 Sep 2023
When I was younger, I once told my dad I was using word to discover all the synonyms to try to write more interestingly.
Many of those words have stayed with me over the decades, such as epitaph, serendipitous, agglomeration, languidly, tartar, preamble, estrafalario (another word for bizarre, in Spanish), among many others.
He replied that it is better to use easier-to-grasp words when possible, most readers wouldn’t fully grasp the context when presented with words they didn’t understand.
The way I took it was: Why use more complicated words to explain something that can be done with simple ones.
I’ve found this same sentence in different forms over the years, in videos about public speaking, writing, making presentations, writing technical documentation, etc.
When I read books that use complex sentence structures, obscure grammar and words, on one hand I feel like the author is being somewhat pompous, showing off their aptitude for words, yet on the other I also feel sometimes when you manage to stumble through a complicated paragraph, over time you learn to see past the air of superiority, and you glean a bit more understanding from the words you aren’t familiar with.
Words that once you might have associated with right-clicking in Word and selecting “Find Synonyms” now have a slightly different meaning, a synonym might convey different intent, it suddenly becomes available to you, to use in contexts where that word might be the exact one that conveys your thoughts at that moment.
You might be aware of the Baader-Meinhof phenomenon, which states that once you learn something you start seeing it everywhere.
I believe a balance can be achieved, where you push the boundaries a tad without sounding pompous, just to inject a bit more colour & texture into one’s language.
The reason I bring this up is that I’ve been reading The Secret History by Donna Tartt, which is littered with exquisite phrases and rare words. The first few chapters also deal with a similar topic, but from the perspective of knowledge, intellectualism, the study of the classics, letting thoughts go wild without restraint, how far can we push it?
I couldn’t help but see the similarities between the theme of the book (so far) and my thoughts about language.
Are you moved by classical music? Does the ending of Spring 1 from Max Richter’s “Reimagining Vivaldi” bring shivers to your spine? I wonder if this highbrow elitism translates equally to music: There are those who see classical music as somewhat pompous; is it possible to enjoy underground rap, hip-hop, hardcore punk, thrash, fusion jazz, and classical music simultaneously? What is the difference between someone who casually enjoys music and one who feels/understands it?
I love most music genres, even those I hated as a kid, such as cumbias. That said, my slight disdain for reggaeton remains strong. I wonder if the same that happens with languages translates to music. Does one who is not well-versed in the intricacies of classical music, the history, the movements, the cadenza, the preludes, the fugues, the counterpoint, the themes, the musical idioms and expressions, would they appreciate music on the same level as someone who’s been classically trained?
Sometimes music touches you beyond the explainable, for example, a complex spicy chord might evoke a certain sensation, such as those during the solo by Cory Henry on the live recording of Lingus by Snarky Puppy. Sometimes when I hear something that evokes such sensations I look around me, unsure if the people I’m with are feeling that same thing, or if they are just passively enjoying it. Is it perceived the same way? Or do we feel music in different ways? Will training lead to allowing that feeling to erupt upon listening to that chord due to our newly found understanding of the underlying play between the notes? The deeper composition? I wonder.
And yet, if you continue down this line of thought… what about people who know so much more… will they think the same? If you could paint in a million tones and textures would you limit yourself to only a few, to make your work more digestible, more humane, more understood? Probably a bad example, but I guess what I’m getting at is:
Would you sabbotage the endless pallette of words at your disposable to conform to a lower-denomination in an effort to be more widely-understood?
26 Sep 2023
More than 15 years ago I wrote a blog post called “how to use 100% of your brain”. It was a bit click-baity as the intention was to illustsrate that we already use 100% all the time, and it included a few references. The intention was mostly to inform people who had grown up with the fables of “we only use a small percentage of our brain, imagine what we could do if we could use all of it?”… I mean, there are even movies on this topic.
Interestingly, it became quite popular, most of my traffic from Google Search went towards that article; I removed it after a while as I hadn’t done proper research on the topic and the information was not very accurate.
I am not a neuroscientist after all, but do you know who is? David Eagleman, author of Livewired.
Note: This book review has some spoilers!
This was a super interesting book to read, quite digestible, with tons of examples.
The premise is that the brain works in a plug-and-play fashion. You can remove sensors, remove body parts, add extra ones, and the brain will adapt… like a general purpose computer. We often talk about plasticity in the brain, but referring to it as livewired makes a lot more sense; it breaks down the traditional myth that “areas of the brain are mapped in a specific way”.
One thing that was a bit of a bummer (in a funny way) was: I had a couple of ideas a few years ago which I was absoluely sure I had invented, and I was just waiting for the right moment to have a few months off some day in the future to work on them, yet here they were in full display on the pages before me, as commercial products I had never heard about.
- My first idea was a sound device for deaf people, using frequencies above 20K Hz, with the idea that even though we can’t process it, the brain might be able to react to those frequencies. The actual product uses lower frequencies (I guess they have verified that higher frequencies had no impact on the cochlea or the vibrations were not ‘converted’ into electrical signals?), yet it works, albeit the sounds are quite audible and for new users it might sound like a wall of sounds. Frequencies and amplitude are used to convey proximity, colours, brightness, etc.
- In the ISS there are sensors built by different countries which not always communicate correctly between each other… In a live-wired system data from sensors and other systems is understood by figuring things out automatically (back-propagation / unsupervised learning>. “Motor” babbling is mentioned (as babbling is the way babies figure out how to communicate, by experimenting out of curiosity). This reminded me of my Curious Actors idea (2015) where AI agents that explore, prod, and learn through curiosity. For example, interacting with APIs or objects in a codebase by performing tasks such as “what happens if I call this function? What happens if I pass this parameter? Does the result change? Are the types constant? What about the structure?”, essentially creating a model of action-reactions to specific events, as well as a dictionary of properties to expect from certain objects (including generalisations as well as specific data for instances of objects). My ultimate idea was that if we could map that virtual world to a real one, the agents would simply collect information from us using an interface, the “objects”, “methods” and “properties” shorthand for interactions with us, physical features and attributes.
A small section of the book read like a sales pitch for one of the products. I feel that was quite unnecessary… a disclosure could have been added instead, or a foreword or afterword with a bit more information on the related commercial interests of the author.
Recap: It is an easy read, not too long, with plenty of examples and illustrations. Some parts dragged on a bit with a few too many examples, but overall a very intriguing book with some really cool concepts! - 8/10
25 Sep 2023
Note: This book review has no spoilers!
This was a great book, I listened to the audio book which, in my opinion, is how this book should be digested. As it is narrated by the voice behind Bob from Bob’s Burgers, the narration style is perfect, the jokes, the sarcasm, the self deprecation, everything carries flawlessly as if listening to Bob on an extended episode.
One of the downside is there is not enough carrying on from one chapter to another, as if isolated small stories, I think more continuity would have helped a bit, however maybe it would have detracted slightly from the comedic narration.
Some chapters were really good, such as those where he tries to get authors to be a part of his book and they give up on him eventually. Also one with the parents in the park where they are judging the other parents and children, and eventually his kid ends up eating excrement.
He narrates everyday situations in such a powerful, excruciating way… taking things that should have ended in a peaceful resolution, dialing it to 11, taking it to the breaking moment, and ending in an explosive climax… sometimes literally. Maybe there were one-too-many ‘shitty’ jokes, at some point I felt myself groveling from my stomach feeling slightly ill.
If you feel like you are failing, reduce your expectations until it no longer feels like that.
I can’t remember if I heard that in the book or if I watched a video where that was mentioned, but I found the quote relatable to this book.
My overall impression was that its a good short book with some very funny moments - 7/10