Well, the Australian year has officially begun (nothing really starts until after Australia Day!) – and knowing that I will need to hand over my intense code duties soon, I spent the week working on some more complex pieces of code.
More specifically, last week I spent a lot of time working on a solution for creating alternate product recommendations usingSpaCy. The idea I had was to measure the Word2Vec distance between product attributes for each attribute a product would have against all attributes of all other products. The problem was though, that this solution is ‘brute force’ and turns out to be in quadratic time without using SpaCy – once I use Word2Vec, it shifts to polynomial time. I need to rework the solution to use a different kind of approach to improve the performance and handle scale. I will tell you what I came up with next week.
At the same time, our Technical Director started playing with my pipeline code – over the weekend he figured out a way to rewrite the entire pipeline so it could be ‘chunked’ and distributed. He simplified and streamlined some of the algorithms and basically threw out a bunch of my code. I felt bad for a few seconds – kind of shows who the better engineer is, right? However, once my ego had its little pity-party, I realised this was awesome – we now have a massively scalable data processing pipeline! I’m really excited to get started on V2 of this platform.
Below are some of the videos I was watching this week…
It’s been a very busy week last week so I did not have as much time to read as I would have liked, so I am still working through the “Learning Computer Architecture with the Raspberry Pi” – which is just such an excellent book.
What else was last week? Well – actually spent some time on less glamorous tasks such as:
- financial modelling to demonstrate
ROI of a piece of data science work we were doing, - high-level project design and scoping
- working on disaster recovery and business continuity planning (which is dry – yet essential!)
- starting to get back into sales planning – this week I am on that in earnest!
- starting to do some deployment planning for new orchestrated container services we are deploying in a few months. (ok, this was actually fun!)
Last week was also very full of Karate training – I really enjoy it – and last week was no exception (other than breaking a toe). Learning the art of Kumite (sparring an opponent) has so many analogs to other areas of life – one of the things I was learning was about classification of the opponent: if you make some broad behavioural categories there are some techniques you can use to identify which category an opponent would ‘fit into’ and if you learn strategies beforehand for dealing with these classes of opponent, you have a vocabulary you can draw upon in an instant.
Think about this, I can use a very simple gesture that does not commit me to an outcome, to better understand my opponent and then reconfigure my attack and defence strategy in a fraction of a second – based on (pretty much) a twitch. This is using categories and response routines to optimise ourOODAloop by using simple signals to ‘orient’ and muscle-memory to ‘act’. In essence, the more this is done the more this becomes subconscious.
The key is using the edge of consciousness to introduce variability at speed into our response (and with the detection mechanisms we use) to throw the pattern matching in our opponent at the same time.
That’s about it for my retrospective… have a great week!
Regards
Joe
