I went for a little trip to Adelaide with my wife for Valentine’s Day, so it was a short week last week – I have to say, I really like Adelaide: People are friendly, stunning landscape, beautiful architecture, excellent food, no ‘Sydney prices’ – and we were there for the opening night of theFringe Festival. What’s not to like?

Solving My Polynomial Time Dilemma

Last week I mentioned I had to change how I do things with my alternate product recommendations project due to the brute force nature of my approach. The problem was that comparing each item with every other item using a neural network (Word2Vec) led to apolynomial time solution. So how can I solve this? I was reflecting on this problem for quite some time and I was on the train conducting a thought experiment when I think I came up with a solution: map everything to agraph topology. Normally, when considering a graph we treat nodes as a homogenous ‘kind of thing’ and edges bind them – instead I am considering nodes represent ‘products’ and nodes also represent ‘attribute-values’; the edges should be the connections of ‘attributes’ to ‘things’ – with multiple connections;

Example:

[Thing 1] -> [Colour:Black] <- [Thing 2]

This means two hops on the graph should get me related products from any given product. Better than that, I should be able to use fast node connectivity approximation algorithms to find related items. I found theNetworkXPython graph library when doing research for the solution. So I will give this a try. It looks quite promising.

Beautiful Soup

I discovered another great Python library last week trying to solve a different problem – it’s calledBeautiful Soup. This brilliant little library helps to parse and navigate web pages (amongst other things) and I was up and running with a proof of concept in an afternoon (as I was rushing out the door to go to the airport!)

Videos I was watching this week…

Performance Reviews

Last week I started working on prep for team performance reviews. I kind of resist doing these as I feel like I intuitively know how everyone is doing, yet, it’s a great way to reflect together and see how far each team-member have come, and to plan a course together for the next quarter that is aligned to our bigger strategic plan. Lesson: Don’t resist! There is a big difference between intuition and a meaningful connection.

I have read enoughJohn Maxwellto know better!

Karate

February is conditioning month!

I broke a toe a couple of weeks ago and it seems like it has affected my Achilles tendon as well – so how do I deal with pain? Read on…

Conditioning training in Karate is an interesting mental exercise: while it causes physical pain, it builds a deep sense of control over one’s self at every level.

Especially since I was already in pain, there were many emotions that came to the surface as I went through the exercises – individually and with a partner. The answer was to just breathe through it and let go…

Mushin no shin – “mind without mind”.

Well, the Australian year has officially begun (nothing really starts until after Australia Day!) – and knowing that I will need to hand over my intense code duties soon, I spent the week working on some more complex pieces of code.

More specifically, last week I spent a lot of time working on a solution for creating alternate product recommendations usingSpaCy. The idea I had was to measure the Word2Vec distance between product attributes for each attribute a product would have against all attributes of all other products. The problem was though, that this solution is ‘brute force’ and turns out to be in quadratic time without using SpaCy – once I use Word2Vec, it shifts to polynomial time. I need to rework the solution to use a different kind of approach to improve the performance and handle scale. I will tell you what I came up with next week.

At the same time, our Technical Director started playing with my pipeline code – over the weekend he figured out a way to rewrite the entire pipeline so it could be ‘chunked’ and distributed. He simplified and streamlined some of the algorithms and basically threw out a bunch of my code. I felt bad for a few seconds – kind of shows who the better engineer is, right? However, once my ego had its little pity-party, I realised this was awesome – we now have a massively scalable data processing pipeline! I’m really excited to get started on V2 of this platform.

Below are some of the videos I was watching this week…

It’s been a very busy week last week so I did not have as much time to read as I would have liked, so I am still working through the “Learning Computer Architecture with the Raspberry Pi” – which is just such an excellent book.

What else was last week? Well – actually spent some time on less glamorous tasks such as:

• financial modelling to demonstrate ROI of a piece of data science work we were doing,
• high-level project design and scoping
• working on disaster recovery and business continuity planning (which is dry – yet essential!)
• starting to get back into sales planning – this week I am on that in earnest!
• starting to do some deployment planning for new orchestrated container services we are deploying in a few months. (ok, this was actually fun!)

Last week was also very full of Karate training – I really enjoy it – and last week was no exception (other than breaking a toe). Learning the art of Kumite (sparring an opponent) has so many analogs to other areas of life – one of the things I was learning was about classification of the opponent: if you make some broad behavioural categories there are some techniques you can use to identify which category an opponent would ‘fit into’ and if you learn strategies beforehand for dealing with these classes of opponent, you have a vocabulary you can draw upon in an instant.

Think about this, I can use a very simple gesture that does not commit me to an outcome, to better understand my opponent and then reconfigure my attack and defence strategy in a fraction of a second – based on (pretty much) a twitch. This is using categories and response routines to optimise ourOODAloop by using simple signals to ‘orient’ and muscle-memory to ‘act’. In essence, the more this is done the more this becomes subconscious.

The key is using the edge of consciousness to introduce variability at speed into our response (and with the detection mechanisms we use) to throw the pattern matching in our opponent at the same time.

That’s about it for my retrospective… have a great week!

Regards
Joe

Writing

Revised theBanking Royal Commissionarticle I had worked on over the previous week. It’s starting to feel like there is more depth. Not totally done, but getting closer. I had a few more ideas about the future of lending compliance that have not quite made it in there yet.

Code

Last week was rather intense on the code front… I rewrote some chunks of code to improve performance and started usingSpaCyfor Natural Language Processing. I was impressed… within an hour I had a working solution for generating keywords and with some refinement I was able to get something to production within a day.

I also spent a lot of time on our build and execution pipelines so that they integrate withAnacondaand make the development process much smoother. I will share this on ourGitHubpage over the next few weeks as I get time.

Reading

Did a ton of reading, but nothing new of note – other than startingGulag Archipelago .

General Thinking

Not much! I was in the zone last week, so not much else to speak of other than getting out the first issue of the Insider Newsletter for the year!