Joe Cincotta

Weekly Journal: Week 7, 2019

I went for a little trip to Adelaide with my wife for Valentine’s Day, so it was a short week last week – I have to say, I really like Adelaide: People are friendly, stunning landscape, beautiful architecture, excellent food, no ‘Sydney prices’ – and we were there for the opening night of theFringe Festival. What’s not to like?

Solving My Polynomial Time Dilemma

Last week I mentioned I had to change how I do things with my alternate product recommendations project due to the brute force nature of my approach. The problem was that comparing each item with every other item using a neural network (Word2Vec) led to apolynomial time solution. So how can I solve this? I was reflecting on this problem for quite some time and I was on the train conducting a thought experiment when I think I came up with a solution: map everything to agraph topology. Normally, when considering a graph we treat nodes as a homogenous ‘kind of thing’ and edges bind them – instead I am considering nodes represent ‘products’ and nodes also represent ‘attribute-values’; the edges should be the connections of ‘attributes’ to ‘things’ – with multiple connections;

Example:

[Thing 1] -> [Colour:Black] <- [Thing 2]

This means two hops on the graph should get me related products from any given product. Better than that, I should be able to use fast node connectivity approximation algorithms to find related items. I found theNetworkXPython graph library when doing research for the solution. So I will give this a try. It looks quite promising.

Beautiful Soup

I discovered another great Python library last week trying to solve a different problem – it’s calledBeautiful Soup. This brilliant little library helps to parse and navigate web pages (amongst other things) and I was up and running with a proof of concept in an afternoon (as I was rushing out the door to go to the airport!)

Videos I was watching this week…

This is an excellent explanation of how cryptography works
A flashback into the dark past of hacking and cracking with an interview with Galahad of the legendary cracking group called Fairlight
John Romero delivers a postmortem on developing the game DOOM

Performance Reviews

Last week I started working on prep for team performance reviews. I kind of resist doing these as I feel like I intuitively know how everyone is doing, yet, it’s a great way to reflect together and see how far each team-member have come, and to plan a course together for the next quarter that is aligned to our bigger strategic plan. Lesson: Don’t resist! There is a big difference between intuition and a meaningful connection.

I have read enoughJohn Maxwellto know better!

Karate

February is conditioning month!

I broke a toe a couple of weeks ago and it seems like it has affected my Achilles tendon as well – so how do I deal with pain? Read on…

Conditioning training in Karate is an interesting mental exercise: while it causes physical pain, it builds a deep sense of control over one’s self at every level.

Especially since I was already in pain, there were many emotions that came to the surface as I went through the exercises – individually and with a partner. The answer was to just breathe through it and let go…

Mushin no shin – “mind without mind”.

Example conditioning exercises.

Weekly Journal: Week 6, 2019

Well, the Australian year has officially begun (nothing really starts until after Australia Day!) – and knowing that I will need to hand over my intense code duties soon, I spent the week working on some more complex pieces of code.

More specifically, last week I spent a lot of time working on a solution for creating alternate product recommendations usingSpaCy. The idea I had was to measure the Word2Vec distance between product attributes for each attribute a product would have against all attributes of all other products. The problem was though, that this solution is ‘brute force’ and turns out to be in quadratic time without using SpaCy – once I use Word2Vec, it shifts to polynomial time. I need to rework the solution to use a different kind of approach to improve the performance and handle scale. I will tell you what I came up with next week.

At the same time, our Technical Director started playing with my pipeline code – over the weekend he figured out a way to rewrite the entire pipeline so it could be ‘chunked’ and distributed. He simplified and streamlined some of the algorithms and basically threw out a bunch of my code. I felt bad for a few seconds – kind of shows who the better engineer is, right? However, once my ego had its little pity-party, I realised this was awesome – we now have a massively scalable data processing pipeline! I’m really excited to get started on V2 of this platform.

Below are some of the videos I was watching this week…

I love this video so much… Just shows how we are so easily caught up in hype of new concepts that are actually founded in old concepts. This video will blow you away once you see how old some of the ‘new ideas’ really are.
Next time anyone says ‘we should be science based and there is no place for spirituality’ get them to watch this video – not because it has anything to do with spirituality – rather, because it shows you how much we know versus how little we actually know why.
Because this guy is just amazing – and it’s beautiful.

It’s been a very busy week last week so I did not have as much time to read as I would have liked, so I am still working through the “Learning Computer Architecture with the Raspberry Pi” – which is just such an excellent book.

What else was last week? Well – actually spent some time on less glamorous tasks such as:

  • financial modelling to demonstrate ROI of a piece of data science work we were doing,
  • high-level project design and scoping
  • working on disaster recovery and business continuity planning (which is dry – yet essential!)
  • starting to get back into sales planning – this week I am on that in earnest!
  • starting to do some deployment planning for new orchestrated container services we are deploying in a few months. (ok, this was actually fun!)

Last week was also very full of Karate training – I really enjoy it – and last week was no exception (other than breaking a toe). Learning the art of Kumite (sparring an opponent) has so many analogs to other areas of life – one of the things I was learning was about classification of the opponent: if you make some broad behavioural categories there are some techniques you can use to identify which category an opponent would ‘fit into’ and if you learn strategies beforehand for dealing with these classes of opponent, you have a vocabulary you can draw upon in an instant.

Think about this, I can use a very simple gesture that does not commit me to an outcome, to better understand my opponent and then reconfigure my attack and defence strategy in a fraction of a second – based on (pretty much) a twitch. This is using categories and response routines to optimise ourOODAloop by using simple signals to ‘orient’ and muscle-memory to ‘act’. In essence, the more this is done the more this becomes subconscious.

The key is using the edge of consciousness to introduce variability at speed into our response (and with the detection mechanisms we use) to throw the pattern matching in our opponent at the same time.

That’s about it for my retrospective… have a great week!

Regards
Joe

Weekly Journal: Week 5, 2019



Writing

Revised theBanking Royal Commissionarticle I had worked on over the previous week. It’s starting to feel like there is more depth. Not totally done, but getting closer. I had a few more ideas about the future of lending compliance that have not quite made it in there yet.

Code

Last week was rather intense on the code front… I rewrote some chunks of code to improve performance and started usingSpaCyfor Natural Language Processing. I was impressed… within an hour I had a working solution for generating keywords and with some refinement I was able to get something to production within a day.

I also spent a lot of time on our build and execution pipelines so that they integrate withAnacondaand make the development process much smoother. I will share this on ourGitHubpage over the next few weeks as I get time.

Reading

Did a ton of reading, but nothing new of note – other than startingGulag Archipelago .

General Thinking

Not much! I was in the zone last week, so not much else to speak of other than getting out the first issue of the Insider Newsletter for the year!



Weekly Journal: Week 4, 2019

Writing

New Article last week – spent some time to get older posts onto Medium and wrote a new one on theBanking Royal Commission

I got back ontoTwitterjust a little… I found some useful conversations are happening there.

Code

In the zone again last week refining some data processing modules and building tools to dynamically create training sets. Working on the final touches of an example project I want to write about – but not there quite yet…

Reading

This week there was some great reading…

Learning Computer Architecture with Raspberry Piby Eben Upton

Artificial Intelligence for Humans, Volume 1: Fundamental AlgorithmsJeff Heaton

Text Classification using Neural Networks

Report on Text Classification using CNN, RNN & HAN

Understand TensorFlow by mimicking its API from scratch

There was a ton more reading on AI, too much to put here – but I can highly recommend Medium as a place to get great articles on AI.

General Thinking

I was struggling for inspiration last week, so I sent an email to a few clients asking them what was on their minds. Not everyone was able to provide a topic – instead there were some spectacularly great ideas for topics and others just told me why they could not give me a topic.

The overwhelming reason people did not give me a topic was their concerns over the outcome of the Banking Royal Commission – and that was my inspiration for the article last week!

Now that the article is out, the first newsletter for 2019 is due out this week!

Planet X2

It was a long weekend in Australia and I finally, finally, got a chance to sit down and play a game for an hour or so. This is a rare occurrence for me indeed. What did I play? A real-time strategy game calledPlanet X2for the Commodore 64 on real vintage hardware. I enjoyed it so much…

Weekly Journal: Week 3, 2019

Last week was fun in a brain-melting kind of way…

Data Science and Neural Networks

I found some great reading on using PyTorch to build neural network classifiers and also a bunch of great writing on using Word2Vec – the problem I was trying to solve was how best to design the inputs and outputs of a NN classifier. There are so many potential data-points that could be used to influence classification, where to start?

I ended up choosing a simple, yet not-quite-so-naive approach of usingTF-IDFto build up a vector that can represent word-embedding after normalising the text-vocabulary withSpaCy. Then we pass in product information to get a predicted product classification (category-confidence vector) …it took a lot of ruminating and reading to get to this conclusion. This week I aim to write the data transformer to build the training set from the data available.

I spent a lot of time last week tuning my fuzzy-match engine as well. As I mentioned last week, an article is in progress. This week I have made the code multi-processor so at this point it is ready for a write up…

Reading

I finished reading theWolfenstein 3D Black Book. This book is basically a code-review of one of the most influential games in history. It was riveting. I could not put it down and basically finished it in a week. Loved every second. There is something really compelling about discovering how a true genius of our time (John Carmack) thinks and solved his programming problems. This is the same reason I used to lovedisassembling demosand games as a kid to understand how visual effects were made.

Goals

Just sharing one of my goals this year is “Fitness every day with a focus on diet”…I will share more goals as it makes sense to. Friday was a 14 hour day at work and the first day I missed doing exercise. I missed not doing exercise after three weeks of sticking to this goal. Got back in the saddle on Saturday and Sunday…


Weekly Journal: Week 2, 2019

Last week was interesting – I had a bunch of things going on at once and all of them were kind of interesting. In fact, it would not be hard to write a full essay on each of these.

Weekly Journal: Week 1, 2019

I decided I would share weekly insights with you as a way to reflect on the week that was. I felt like there is so much we do every week that could be discussed, shared, considered further. I wanted a place for that to happen and so this is it…

lod-cloud

Getting Data in the Enterprise

So many organisations today struggle with their data. The world of big data, analytics and predictive modelling is so attractive, yet, the perceived complexity of achieving a data nirvana is so great that data projects either spiral out of control or never start. So how can you get a handle on your current data, whip it into shape and start using it?

7670055210_ceb0c9ef9a_b

Managing Complexity

No matter what industry you’re in, you have to deal with complexity on a daily basis. From the complexity of computer systems you use to get work done, to the complexity of the projects you’re managing – it seems a byproduct of the information age is complexity.

thinking.studio

Level 25, 88 Phillip Street
Sydney
NSW 2000
Australia

+61 2 8024 5975
[email protected]

©2003-2019 Pixolut Pty Ltd trading as Thinking.Studio
All Rights Reserved