<![CDATA[Rico Meinl - Rico Meinl]]>https://rmeinl.com/https://rmeinl.com/favicon.pngRico Meinl - Rico Meinlhttps://rmeinl.com/Ghost 4.3Mon, 26 Apr 2021 11:34:49 GMT60<![CDATA[How to set up your own ENS domain name]]>https://rmeinl.com/how-to-set-up-ens-domain-name/60857c71d3d3ed25945ea735Sun, 25 Apr 2021 16:11:36 GMTThis is a short tutorial on how to set up your own ENS domain name using MetaMask and Google Chrome. Normally I'm using Brave but I thought doing the demos in Chrome would allow more people to access it.

  1. Go to https://ens.domains/ and click Launch App.

2. Enter the name you are planning to register. It can end with .eth but doesn't have to.

3. Now we have to connect ENS to an Ethereum account. Though they offer multiple ways to connect your wallet (as shown below) we are going to use MetaMask for this tutorial. If you already have a MetaMask account set up, you can jump straight to step 10.

4. In order to download MetaMask, go to https://metamask.io/download.html and press Install MetaMask for Chrome. You'll be redirected to the Chrome Web Store.

5. Next, add MetaMask to Chrome by pressing Add to Chrome. After installation, you'll be redirected to the setup screen.

6. Press Get Started and Create a Wallet unless you already have one.

7. Create a password for your wallet.

8. Store your secret backup phrase in a safe place. It makes it easy to back up and restore your account. (The only reason I'm showing my phrase is because I'm using a throwaway account for this tutorial. You should never show it to anyone.)

9. You're all set. You should now see a page with your MetaMask account like this.

10. Now that MetaMask is all set up, switch back to the ENS tab and click Connect to connect with your wallet. It'll open the same window as in step 3, but it should also include MetaMask now. You might have to refresh your browser.

11. Click MetaMask and select the account you want to authenticate with. Click Next and finally Connect. On the left side next to the ENS domain name you should now see that your account is connected to the mainnet.

12. In order to pay for the domain name you need to add some Ether to your account. The fastest way to do that is a direct deposit as shown below. Go to Buy > Directly Deposit Ether > View Account to get your MetaMask Ether address. Use the wallet of your choice to send Ether to this account. It could take up to 10 minutes for your funds to arrive.

13. Click on the domain name you want to register and select the number of years you want to reserve it for (2+ years are recommended, given the gas fees). There are three steps in total as listed on the website.

14. Request to register: Your wallet will open and you will be asked to confirm the first of two transactions required for registration.

15. Wait for 1 minute: The waiting period is required to ensure another person hasn’t tried to register the same name and protect you after your request. Afterward, your screen should look like this.

16. Complete Registration: Click Register and your wallet will re-open. Only after the 2nd transaction is confirmed you'll know if you got the name. This could take up to 10 minutes. As you can see, this transaction cost me about $50 in total but the gas fees are variable so it might be more or less depending on when you submit yours.

17. After the registration is completed, you should see the name show up under My Account.

18. Click Reverse record: not set. Select your ENS name then click Save, and submit the transaction to save it on the blockchain.

19. After about 10 minutes you should see that your reverse record has been set up successfully.

20. In order to add some records, click on your name in the list below. You should see that it already points to your Ethereum address. Click on Add/Edit Record. I'm going to add my BTC address, my website and twitter as well as github handle.

21. Finally, confirm the transaction and submit it to the blockchain via MetaMask. This should take another 10 minutes.

22. That's it! You can now use a browser like Opera or Brave to check whether everything worked out. I'm using Brave here, which will initially ask for confirmation to redirect via ENS. You should then see your record. If you neither have Brave or Opera, just go to https://app.ens.domains/name/<your_domain>.eth


We just walked through how to set up your own ENS domain name using MetaMask and Chrome. To give you a rough idea about the costs, the whole process cost me $97.86. Here's the breakdown:

  • $6.17 for step 14 (initial request)
  • $46.26 for step 16 (paying for the name)
  • $21.48 for step 18 (setting up the reverse record)
  • $23.95 for step 21 (adding custom records)

Obviously the majority of these costs are gas fees, you only pay ENS for step 16 so it will vary for you depending on when you set up yours.

Hope this was helpful!

]]>
<![CDATA[Setting up Virtual Flow on AWS using Parallelcluster and Slurm]]>This is a short tutorial on how to set up AWS Parallelcluster with Slurm to run VirtualFlow.

VirtualFlow is a versatile, parallel workflow platform for carrying out virtual screening related tasks on Linux-based computer clusters of any type and size which are managed by a batchsystem (such as SLURM).

AWS

]]>
https://rmeinl.com/setting-up-virtual-flow-on-aws/6085751174348916f1b089a8Mon, 19 Apr 2021 10:27:48 GMTThis is a short tutorial on how to set up AWS Parallelcluster with Slurm to run VirtualFlow.

VirtualFlow is a versatile, parallel workflow platform for carrying out virtual screening related tasks on Linux-based computer clusters of any type and size which are managed by a batchsystem (such as SLURM).

AWS Parallelcluster with Slurm

Creating our working environment

First, we'll create our working directory and set up a virtual environment using poetry. We need to add the awscli package as well as the aws-parallelcluster package.

mkdir parallel_cluster
cd parallel_cluster
poetry init
poetry add awscli aws-parallelcluster

Setting up the cluster config

To set up the AWS Parallelcluster I mainly followed this post. We start by creating the config for our cluster. Make sure to create an EC2 key pair beforehand.

 $ poetry run pcluster configure                  
Allowed values for AWS Region ID:
1. ap-northeast-1
2. ap-northeast-2
3. ap-south-1
4. ap-southeast-1
5. ap-southeast-2
6. ca-central-1
7. eu-central-1
8. eu-north-1
9. eu-west-1
10. eu-west-2
11. eu-west-3
12. sa-east-1
13. us-east-1
14. us-east-2
15. us-west-1
16. us-west-2
AWS Region ID [us-west-2]: 16
Allowed values for EC2 Key Pair Name:
1. parallelcluster
EC2 Key Pair Name [parallelcluster]: 1
Allowed values for Scheduler:
1. sge
2. torque
3. slurm
4. awsbatch
Scheduler [slurm]: 3
Allowed values for Operating System:
1. alinux
2. alinux2
3. centos7
4. centos8
5. ubuntu1604
6. ubuntu1804
Operating System [alinux2]: 2
Minimum cluster size (instances) [0]: 1
Maximum cluster size (instances) [10]: 
Head node instance type [t2.micro]: c4.large
Compute instance type [t2.micro]: c4.xlarge
Automate VPC creation? (y/n) [n]: y

We should now have a config file similar to this:

$ cat ~/.parallelcluster/config 
[aws]
aws_region_name = us-west-2

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}

[global]
cluster_template = default
update_check = true
sanity_check = true

[vpc default]
vpc_id = vpc-*****************
master_subnet_id = subnet-*****************

[cluster default]
key_name = parallelcluster
scheduler = slurm
master_instance_type = c4.large
base_os = alinux2
vpc_settings = default
queue_settings = compute

[queue compute]
enable_efa = false
enable_efa_gdr = false
compute_resource_settings = default

[compute_resource default]
instance_type = c4.xlarge
min_count = 1

Creating the cluster

After the config file is set, we can create our cluster using the following commands. AWS will then spin up our CloudFormation stack which will take a couple of minutes.

$ poetry run pcluster create test-cluster
Beginning cluster creation for cluster: test-cluster
Creating stack named: parallelcluster-test-cluster
...

In order to access our head node we can run the following:

poetry run pcluster ssh test-cluster -i ~/.ssh/<key_name>

VirtualFlow

To get started with VirtualFlow I recommend running through the first tutorial to make sure the cluster has been set up correctly. I'm only going through the changes that need to be made and list the other steps solely for completeness. The tutorial does a good job at explaining each individual step.

Setting up VirtualFlow

First, we download the tutorial files and unzip them.

$ wget https://virtual-flow.org/sites/virtual-flow.org/files/tutorials/VFVS_GK.tar
$ tar -xvf VFVS_GK.tar
$ cd VFVS_GK/tools

Preparing the config files

There are two files in which we need to make changes. We want to make sure our batchsystem is set to 'SLURM' and change the partition to 'compute' which is the default name when we use AWS Parallelcluster.

# tools/templates/all.ctrl
...
batchsystem=SLURM
# Possible values: SLURM, TOQRUE, PBS, LSF, SGE
# Settable via range control files: No
...
partition=compute
# Partitions are also called queues in some batchsystems
# Settable via range control files: Yes

If 'compute' doesn't work, try running the following command to retrieve the correct partition name:

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST 
compute*     up   infinite      5  idle~ compute-dy-c4xlarge-[5-9] 
compute*     up   infinite      5  alloc compute-dy-c4xlarge-[1-4],compute-st-c4xlarge-1 

The second config file we need to adjust is the Slurm job template script. Usually we should be able to leave all the default values but I ran into this error:

srun: error: Unable to create step for job 874794: Memory required by task is not available

In order to solve it, we simply comment out the line with the --mem-per-cpu parameter.

# Slurm Settings
###############################################################################

#SBATCH --job-name=h-1.1
##SBATCH --mail-user=To be completed if uncommented
#SBATCH --mail-type=fail
#SBATCH --time=00-12:00:00
##SBATCH --mem-per-cpu=1024M
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --partition=main
#SBATCH --output=../workflow/output-files/jobs/job-1.1_%j.out           # File to which standard out will be written
#SBATCH --error=../workflow/output-files/jobs/job-1.1_%j.out            # File to which standard err will be written
#SBATCH --signal=10@300

As a last preparation step we simply go back to the /tools subfolder and run this command:

./vf_prepare_folders.sh

More details here: https://docs.virtual-flow.org/tutorials/-LdE94b2AVfBFT72zK-v/vfvs-tutorial-1/setting-up-the-workflow.

Starting the jobs

To spin up our nodes, we simply run this command:

./vf_start_jobline.sh 1 12 templates/template1.slurm.sh submit 1

More details can be found here: https://docs.virtual-flow.org/tutorials/-LdE94b2AVfBFT72zK-v/vfvs-tutorial-1/starting-the-workflow.

Monitoring and Wrapping Up

In order to monitor the jobs and view the files after completion, I recommend the respective sections of the tutorial:

Monitoring

Completed Workflow

Using our own files

Running the same workflow with our own files is pretty straightforward. After we downloaded the template files in the 'Setting up VirtualFlow' step we need to replace the ligand library as well as our target protein.

Replacing the ligand library

The second tutorial in the VirtualFlow documentation has a section dedicated to this.

Using a different Protein

Here, I downloaded AutoDock Vina together with MGLTools and followed the tutorial on http://vina.scripps.edu which looks outdated but still works fine. We can use AutoDock Vina to convert our protein from .pbd to .pdbqt and use the 'GridBox' tool to get the necessary parameters for respective receptor config file.

# ../input-files/smina_rigid_receptor1/config.txt
receptor = ../input-files/receptor/<protein>.pdbqt
center_x = 28.614
center_y = 15.838
center_z = -2.045
size_x = 36.0
size_y = 32.0
size_z = 36.0
exhaustiveness = 4
scoring = vinardo
cpu = 1

We add our protein to the folder and change both the smina (/input-files/smina_rigid_receptor1) and qvina receptor (/input-files/qvina02_rigid_receptor1) config files.

That's it. Now we can follow the rest of the steps outlined in the 'VirtualFlow' section above.

]]>
<![CDATA[The 80/20 Computer Science Degree]]>https://rmeinl.com/nand2tetris/6085751174348916f1b089a7Tue, 30 Mar 2021 22:12:56 GMTDRAFT/DISCLAIMER — This post is a submission to a competition on 1729.com. No prizes will be awarded for any submissions at this time. Learn more at 1729.com/decentralized-task-creation.

Nonetheless, I highly recommend everyone who is interested in CS to take this course.

Listening to podcasts with people in tech, you'll often hear that they got interested in the field because they built their own computers or coded their own games. Elon, for example, sold his first computer game at the age of 12 and built custom computers for others in university.

Now, that most people have laptops it becomes harder to just open them up, check what's inside and put it back together. Of course you could buy go and buy all the parts separately or get a DIY kit. Though this might not be logistically feasible for everyone which is a shame because this kind of tinkering is a great learning vehicle for anything related to Computer Science.

What if you could virtualize the whole experience while being guided by some world-class CS professors? Enter Nand to Tetris.

A game changer in CS education

Nand to Tetris was created by two CS professors, Noam Nisan and Shimon Schocken. In a nutshell, you'll build your own computer in a bottom-up fashion all the way up from NAND gates.
In the process, you'll get a hands-on coverage of most of the important ideas and techniques in applied computer science, focusing on computer architecture, compilation, and software engineering, in one course. Nand to Tetris also provides a hands-on overview of key data structures and algorithms, as they unfold in the context of 12 captivating hardware and software development projects.

Nand to Tetris courses are now taught at 200+ universities and high schools around the world. The students who take them range from high school students to Ph.D. students to Google engineers.

Task: Earn $500 in BTC

  1. Boolean Logic
  2. Boolean Arithmetic
  3. Sequential Logic
  4. Machine Language
  5. Computer Architecture
  6. Assembler
  7. Virtual Machine I: Stack Arithmetic
  8. Virtual Machine II: Program Control
  9. High Level Language
  10. Compiler I: Syntax Analysis
  11. Compiler II: Code Generation
  12. Operating System
  13. Tetris

During these 12 projects you will build your own Assembler, Virtual Machine, Java-like High Level Language, Compiler and Operating System. In the optional 13th project you can tie all these things together to write an implementation of Tetris or any other game of your choice using all the components you previously built.

There is a guided Coursera course with two parts but just using the links above or the book works perfectly fine. Check out the introduction video here. Some inspirational projects can be found here.

]]>
<![CDATA[Sequential and simultaneous modes of awareness]]>The most interesting part in Ted Chiang's "Story of your life" is the parallel of the causal and teleological explanation with a sequential and simultaneous mode of awareness.

Fermat's principle of least time can be interpreted in terms of cause and effect: a difference

]]>
https://rmeinl.com/sequential-and-simultaneous-modes-of-awareness/6085751174348916f1b089a3Mon, 30 Nov 2020 17:24:24 GMTThe most interesting part in Ted Chiang's "Story of your life" is the parallel of the causal and teleological explanation with a sequential and simultaneous mode of awareness.

Fermat's principle of least time can be interpreted in terms of cause and effect: a difference in the index of refraction caused the light ray to change direction when it hit the surface of the water. This is most intuitive to us humans.

It can also be interpreted teleologically: the ray of light has to know where its destination is in order to compute the path of least time. This is more intuitive to the heptapods.

The parallel to the causal explanation is a sequential mode of awareness: experiencing events in order, and perceiving their relationship as cause and effect. This is how humans experience things. We don't know the future and are therefore able to exercise free will.

The parallel to the teleological explanation is a simultaneous mode of awareness: experiencing events all at once, and perceiving a purpose underlying them all. This is how heptapods experience. They already know the future, so freedom is meaningless and every act is performative*.

If you have free will, it's impossible to know about the future because you could change it. On the other side, if you know the future you cannot act freely anymore. (as in the example of the book of ages).

Sequential and simultaneous modes of awareness are like the optical illusion of the old and young lady. Both are valid but you can't see them at the same time.


*Performative language: Saying equals doing.
Example: At a wedding ceremony everybody knows that at the end the pastor will pronounce the couple husband and wife but it doesn't count until he actually says it.

]]>
<![CDATA[The best way to encompass the future is by building a strong set of beliefs.]]>Using claims as a first-class citizen in your thinking helps you move towards strong beliefs.

If you don't explicitly state your claims you're never going to move in any direction. Everything will seem kind of relevant and worth pursuing.

Writing down claims can help manifest them.

]]>
https://rmeinl.com/the-best-way-to-encompass-the-future-is-by-building-a-strong-set-of-beliefs/6085751174348916f1b089a2Wed, 28 Oct 2020 22:13:50 GMTUsing claims as a first-class citizen in your thinking helps you move towards strong beliefs.

If you don't explicitly state your claims you're never going to move in any direction. Everything will seem kind of relevant and worth pursuing.

Writing down claims can help manifest them. It helps to understand their implications, as well as supporting and opposing claims.

Claims eventually turn into beliefs and beliefs give perspective. They act like gravity. Strong beliefs are something that new information can be attached to.

Related:

The best way to get to a good forecast is by making predictions with limited information and trying to find opposing evidence; then using the accumulated insights to improve your predictions.
Discussed on Twitter.
]]>
<![CDATA[Why we gain compounding benefits from incremental knowledge tools]]>Knowledge and productivity are like compound interest. As knowledge workers, we live on the margins and every seemingly little improvement can add up to that compound in the long run.

The more you know, the more you learn; the more you learn, the more you can do; the more you

]]>
https://rmeinl.com/the-marginal-benefits-we-gain-from-knowledge-tools-are/6085751174348916f1b089a1Tue, 27 Oct 2020 11:43:49 GMTKnowledge and productivity are like compound interest. As knowledge workers, we live on the margins and every seemingly little improvement can add up to that compound in the long run.

The more you know, the more you learn; the more you learn, the more you can do; the more you can do, the more the opportunity.

With the old file cabinet like note taking systems there was literally no gain when going from 10 notes to 10.000 notes. It was probably more of a downward linear trend because of the growing lack of structure. With graph-based tools like Roam Research, your knowledge management system can improve almost exponentially the more you add to it (if done right). The increasing number of notes allows for ever more unexpected connections.

Roam Research is also an IDE for knowledge work and enables us to treat notes as composable blocks of knowledge. Text is not as composable as code or graphic elements.

But as the Zettelkasten shows, the notes that contribute to an idea and eventually to a piece of content are very much composable. Knowledge systems that compose and have atomic statements make it much easier to write and publish.

The interface of Roam is mouldable and we can build our own meta-tools on top of it. The question for all the builders will be if we can make the new meta-tools for knowledge as valuable as the meta-tools for programming.

Discussed on Twitter.
]]>
<![CDATA[Embed Twitter Threads in Roam Research]]>https://rmeinl.com/embed-tweet-threads-in-roam-research/6085751174348916f1b089a0Fri, 23 Oct 2020 18:50:23 GMT

Get it here

How to use:
- Paste a tweet url into Roam.
- The thread is then copied to your clipboard.
- Paste it into Roam via CMD+V (Mac) CTRL-V (Windows).

]]>
<![CDATA[Recommender Systems: The Most Valuable Application of Machine Learning (Part 2)]]>https://rmeinl.com/recommender-systems-the-most-valuable-application-of-machine-learning-part-2/6085751174348916f1b0899eSun, 04 Oct 2020 20:50:12 GMTWhy Recommender Systems are the most valuable application of Machine Learning and how Machine Learning-driven Recommenders already drive almost every aspect of our lives.

Read this article on Medium.

Recommender Systems already drive almost every aspect of our daily lives.

This is the second part of the article published on 11 May. In the first part I covered:

  • Business Value
  • Problem Formulation
  • Data
  • Algorithms

In this second part I will cover the following topics:

  • Evaluation Metrics
  • User Interface
  • Cold-start Problem
  • Exploration vs. Exploitation
  • The Future of Recommender Systems

Throughout this article, I will continue to use examples of the companies that have built the most widely used systems over the last couple of years, including Airbnb, Amazon, Instagram, LinkedIn, Netflix, Spotify, Uber Eats, and YouTube.


Evaluation Metrics

Now that we have the algorithm for our Recommender System, we need to find a way to evaluate its performance. As with every Machine Learning model, there are two types of evaluation:

  1. Offline Evaluation
  2. Online Evaluation
Offline/Online Testing Framework

Generally speaking, we can consider the Offline Evaluation metrics as low-level metrics, that are usually easily measurable. The most well-known example would be Netflix choosing to use root mean squared error (RMSE) as a proxy metric for their Netflix Prize Challenge. The Online Evaluation metrics are the high-level business metrics that are only measurable as soon as we ship our model into the real world and test it with real users. Some examples include customer retention, click-through rate, or user engagement.

Offline Evaluation

As most of the existing Recommender Systems consist of two stages (candidate generation and ranking), we need to pick the right metrics for each stage. For the candidate generation stage, YouTube, for instance, focuses on high precision so “out of all the videos that were pre-selected how many are relevant”. This makes sense given that in the first stage we want to filter for a smaller set of videos whilst making sure all of them are potentially relevant to the user. In the second stage, presenting a few “best” recommendations in a list requires a fine-level representation to distinguish relative importance among candidates with high recall (“how many of the relevant videos did we find”).

Often, most of the examples are using the standard evaluation metrics used in the Machine Learning community: from ranking measures, such as normalized discounted cumulative gain, mean reciprocal rank, or fraction of concordant pairs, to classification metrics including accuracy, precision, recall, or F-score.

Instagram formulated the optimization function of their final pass model a little different:

We predict individual actions that people take on each piece of media, whether they’re positive actions such as like and save, or negative actions such as “See Fewer Posts Like This” (SFPLT). We use a multi-task multi-label (MTML) neural network to predict these events.

As appealing as offline experiments are, they have a major drawback: they assume that members would have behaved in the same way, for example, playing the same videos, if the new algorithm being evaluated had been used to generate the recommendations. That’s why we need online evaluation to measure the actual impact our model has on the higher-level business metrics.

Online Evaluation

The approach to be aware of here is A/B testing. There are many interesting and exhaustive articles/courses that cover this well, therefore I won’t spend too much time on this. The only slight variation I have encountered is Netflix’s approach called “Consumer Data Science” that you can read about it here.

The most popular high-level metrics that companies are measuring here are Click-Through Rate and Engagement. Uber Eats goes further here and designed a multi-objective tradeoff that captures multiple high-level metrics to account for the overall health of their three-sided marketplace (among others: Marketplace Fairness, Gross Bookings, Reliability, Eater Happiness). In addition to medium-term engagement, Netflix focuses on member retention rates as their online tests can range from between 2–6 months.

YouTube famously prioritizes watch-time over click-through rate. They even wrote an article, explaining why:

Ranking by click-through rate often promotes deceptive videos that the user does not complete (“clickbait”) whereas watch time better captures engagement

Evaluating Embeddings

As covered in the section on algorithms, embeddings are a crucial part of the candidate generation stage. However, unlike with a classification or regression model, it’s notoriously difficult to measure the quality of an embedding given that they are often being used in different contexts. A sanity check we can perform is to map the high-dimensional embedding vector into a lower-dimensional representation (via PCA, t-SNE, or UMAP) or apply clustering techniques such as k-means and then visualize the results. Airbnb did this with their listing embeddings to confirm that listings from similar locations are clustered together.


User Interface

For a Machine Learning Engineer or Data Scientist, the probably most overlooked aspect of the equation is the User Interface. The problem is that if your UI does not contain the needed components to showcase the recommendations or showcases them in the wrong context, the feedback loop is inherently flawed.

Let’s take Linkedin as an example to illustrate this. If I’m browsing through people’s profiles, on the right-hand side of the screen I see recommendations for similar people. When I’m browsing through companies, I see recommendations for similar companies. The recommendations are adapted to my current goals and context and encourage me to keep browsing the site. If the similar companies recommendations would appear on a person’s profile, I would probably be less encouraged to click on their profile as it is not what I am currently looking for.

Similar User Recommendations on Linkedin
Similar Companies Recommendations on Linkedin

You can build the best Recommender System in the world, however, if your interface is not designed to serve the user’s needs and wants, no one will appreciate the recommendations. In fact, the User Interface challenge is so crucial that Netflix turned all components on their website into dynamic ones which are assembled by a Machine Learning algorithm to best reflect the goals of a user.

Spotify followed that model and adopted a similar layout for their home screen design, as can be seen below.

Personalizing Spotify Home with Machine Learning (Source: Spotify)

This is an ongoing area where there is still a lot of experimentation. As an example, YouTube recently changed their homepage interface to enable users to narrow down the recommendations for different topics:

New YouTube Home Page

Cold-start Problem

The cold-start problem is often seen in Recommender Systems because methods such as collaborative filtering rely heavily on past user-item interactions. Companies are confronted with the cold-start problem in two ways: user and item cold-start. Depending on the type of platform, either one of them is more prevalent.

User cold-start

Imagine a new member signs up for Netflix. At this point, the company doesn’t know anything about the new members’ preferences. How does the company keep her engaged by providing great recommendations?

In Netflix’s case, new members get a one-month free trial, during which cancellation rates are the highest while they decrease quickly after that. This is why any improvements to the cold-start problem present an immense business opportunity for Netflix, in order to increase engagement and retention in those first 30 days. Today, their members are given a survey during the sign-up process, during which they are asked to select videos from an algorithmically populated set that is then used as an input into all of their algorithms.

Item cold-start

Companies face a similar challenge when new items or content are added to the catalog. Platforms like Netflix or Prime Video hold an existing catalog of media items that changes less frequently (it takes time to create movies or series!), therefore they struggle less with this. On the contrary, on Airbnb or Zillow, new listings are created every day and at that point, they do not have an embedding as they were not present during the training process. Airbnb solves this the following way:

To create embeddings for a new listing we find 3 geographically closest listings that do have embeddings, and are of same listing type and price range as the new listing, and calculate their mean vector.

For Zillow, this is especially critical as some of the new home listings might only be on the site for a couple of days. They creatively solved this problem by creating a neural network-based mapping function from the content space to the embedding space, which is guided by the engagement data from users during the learning phase. This allows them to map a new home listing to the learned embedding space just by using its features.


Exploration vs. Exploitation

The concept of exploration/exploitation can be seen as the balancing of new content with well-established content. I was going to illustrate this concept myself, while I found this great excerpt that hits it right out of the ballpark:

“Imagine you’ve just entered an ice cream shop. You now face a crucial decision — out of about 30 flavors you need to choose only one!
You can go with two strategies: either go with that favorite flavor of yours that you already know is the best; or explore new flavors you never tried before, and maybe find a new best flavor.
These two strategies — exploitation and exploration — can also be used when recommending content. We can either exploit items that have high click-through rate with high certainty — maybe because these items have been shown thousands of times to similar users, or we can explore new items we haven’t shown to many users in the past. Incorporating exploration into your recommendation strategy is crucial — without it, new items don’t stand a chance against older, more familiar ones.”

(Source: Recommender Systems: Exploring the Unknown Using Uncertainty)

This tradeoff is a typical reinforcement learning problem and a commonly used approach is the multi-armed bandit algorithm. This is used by Spotify for the personalization of each users’ home page as well as Uber Eats for personalized recommendations optimized for their three-sided marketplace. Two scientists at Netflix gave a great talk about how they are using the MAB framework for movie recommendations.

Though I should mention that this is, by no means, the final solution to this problem, it seems to work for Netflix, Spotify, and Uber Eats, right?

Yes. But!

Netflix has roughly 160 million users and about 6.000 movies/shows. Spotify has about 230 million users and 50 million songs + 500.000 podcasts.

Twitter’s 330 million active users generate more than 500 million tweets per day (350.000 tweets per minute, 6.000 tweets per second). And then there’s YouTube, with its 300 hours of videos uploaded every minute!

The exploration space in the two latter cases is a little bit bigger than in the case of Netflix or Uber Eats, which makes the problem a lot more challenging.


The Future of Recommender Systems

This is the end of my little survey over Recommender Systems. As we have observed, Recommender Systems already guide so many aspects of our life. All the algorithms we covered over the course of these two articles are competing for our attention every day. And after all, they are all maximizing the time spent on their platform. As I illustrated in the section on Evaluation methods, most of the algorithms are optimizing for something like Click-through rate, engagement, or in YouTube’s case: watch time.

What does that mean for us as a consumer?

What it means is, that we are not in control of our desires anymore. While this might sound poetic, think about it. Let’s look at YouTube; we all have goals when coming to the site. We might want to listen to music, watch something funny, or learn something new. But all the content that is recommended to us (either through the Home Page recommendations, Search Ranking, or Watch Next) is optimized to keep us on the site for longer.

Lex Fridman and François Chollet had a great conversation about this on the Artificial Intelligence Podcast. Instead of choosing the metric to optimize for, what if companies would put the user in charge of choosing their own objective function? What if they would take the personal goals of the user’s profile into account and ask the user, what do you want to achieve? Right now, this technology is almost like our boss and we’re not in control of it. Wouldn’t it be incredible to leverage the power of Recommender Systems to be more like a mentor, a coach, or an assistant?

Imagine, as a consumer, you could ask YouTube to optimize the content to maximize learning outcomes. The technology is certainly already there. The challenge would really lie in aligning this with the existing business models and designing the right interface to empower the user to make that choice, and also to change as their goals evolve. With its new interface, YouTube is perhaps already taking baby-steps in that direction by putting the user in charge to select categories that she wants to see recommendations for. But this is just the beginning.

Could this be the way forward or is this just a consumer’s dream?


Resources

François Chollet: Keras, Deep Learning, and the Progress of AI | Artificial Intelligence Podcast

Airbnb — Listing Embeddings in Search Ranking

Airbnb — Machine Learning-Powered Search Ranking of Airbnb Experiences

Amazon — Amazon.com Recommendations Item-to-Item Collaborative Filtering

Amazon — The history of Amazon’s recommendation algorithm

Instagram — Powered by AI: Instagram’s Explore recommender system

LinkedIn — The Browsemaps: Collaborative Filtering at LinkedIn

Netflix — Netflix Recommendations: Beyond the 5 stars (Part 1)

Netflix — Netflix Recommendations: Beyond the 5 stars (Part 2)

Netflix — The Netflix Recommender System: Algorithms, Business Value, and Innovation

Netflix — Learning a Personalized Homepage

Pandora — Pandora’s Music Recommender

Spotify — Discover Weekly: How Does Spotify Know You So Well?

Spotify — For Your Ears Only: Personalizing Spotify Home with Machine Learning

Spotify — From Idea to Execution: Spotify’s Discover Weekly

Twitter — Embeddings@Twitter

Uber Eats — Food Discovery with Uber Eats: Recommending for the Marketplace

Uber Eats — Food Discovery with Uber Eats: Using Graph Learning to Power Recommendations

YouTube — The YouTube Video Recommendation System

YouTube — Collaborative Deep Learning for Recommender Systems

YouTube — Deep Neural Networks for YouTube Recommendations

Zillow — Home Embeddings for Similar Home Recommendations

Andrew Ng’s Machine Learning Course (Recommender Systems)

Google’s Machine Learning Crash Course — Embeddings


]]>
<![CDATA[Recommender Systems: The Most Valuable Application of Machine Learning (Part 1)]]>https://rmeinl.com/recommender-systems-the-most-valuable-application-of-machine-learning-part-1/6085751174348916f1b0899dSun, 04 Oct 2020 20:46:09 GMTWhy Recommender Systems are the most valuable application of Machine Learning and how Machine Learning-driven Recommenders already drive almost every aspect of our lives.

Read this article on Medium.

Recommender Systems already drive almost every aspect of our daily lives.

Look back at your week: a Machine Learning algorithm determined what songs you might like to listen to, what food to order online, what posts you see on your favorite social networks, as well as the next person you may want to connect with, what series or movies you would like to watch, etc…

Machine Learning already guides so many aspects of our life without us necessarily being conscious of it. All of the applications mentioned above are driven by one type of algorithm: recommender systems.

In this article, I will explore and dive deeper into all the aspects that come into play to build a successful recommender system. The length of this article got a little out of hand so I decided to split it into two parts. This first part will cover:

  • Business Value
  • Problem Formulation
  • Data
  • Algorithms

The Second Part will cover:

  • Evaluation Metrics
  • User Interface
  • Cold-start Problem
  • Exploration vs. Exploitation
  • The Future of Recommender Systems

Throughout this article, I will be using examples of the companies that have built the most widely used systems over the last couple of years, including Airbnb, Amazon, Instagram, LinkedIn, Netflix, Spotify, Uber Eats, and YouTube.


Business Value

Harvard Business Review made a strong statement by calling Recommenders the single most important algorithmic distinction between “born digital” enterprises and legacy companies. HBR also described the virtuous business cycle these can generate: the more people use a company’s Recommender System, the more valuable they become and the more valuable they become, the more people use them.

The Virtuous Business Cycle of Recommender Systems (source: MDPI, CC)

We are encouraged to look at recommender systems, not as a way to sell more online, but rather to see it as a renewable resource for relentlessly improving customer insights and our own insights as well. If we look at the illustration above, we can see that many legacy companies also have tons of users and therefore tons of data. The reason their virtuous cycle has not picked up as much as the ones off Amazon, Netflix or Spotify is because of the lack of knowledge on how to convert their user data into actionable insights, which can then be used to improve their product or services.

Looking at Netflix, for example, shows how crucial this is, as 80% of what people watch comes from some sort of recommendation. In 2015, one of their papers quoted:

“We think the combined effect of personalization and recommendations save us more than $1B per year.”

If we look at Amazon, 35% of what customers purchase at Amazon comes from product recommendations and at Airbnb, Search Ranking and Similar Listings drive 99% of all booking conversions.


Problem Formulation

Now that we’ve seen the immense value, companies can gain from Recommender Systems, let’s look at the type of challenges that can be solved by them. Generally speaking, tech companies are trying to recommend the most relevant content to their users. That could mean:

  • similar home listings (Airbnb, Zillow)
  • relevant media, e.g. photos, videos and stories (Instagram)
  • relevant series and movies (Netflix, Amazon Prime Video)
  • relevant songs and podcasts (Spotify)
  • relevant videos (YouTube)
  • similar users, posts (LinkedIn, Twitter, Instagram)
  • relevant dishes and restaurants (Uber Eats)

The formulation of the problem is critical here. Most of the time, companies want to recommend content that users are most likely to enjoy in the future. The reformulation of this problem, as well as the algorithmic changes from recommending “what users are most likely to watch” to “what users are most likely to watch in the futureallowed Amazon PrimeVideo to gain a 2x improvement, a “once-in-a-decade leap” for their movie Recommender System.

“Amazon researchers found that using neural networks to generate movie recommendations worked much better when they sorted the input data chronologically and used it to predict future movie preferences over a short (one- to two-week) period.”

Data

Recommender Systems usually take two types of data as input:

  • User Interaction Data (Implicit/Explicit)
  • Item Data (Features)

The “classic”, and still widely used approach to recommender systems based on collaborative filtering (used by Amazon, Netflix, LinkedIn, Spotify and YouTube) uses either User-User or Item-Item relationships to find similar content. I’m not going to go deeper into the inner workings of this, as there are a lot of articles on that topic — like this one — that explain this concept well.

The user interaction data is the data we gather from the weblogs and can be divided into two groups:

Explicit data: explicit input from our users (e.g. movie ratings, search logs, liked, commented, watched, favorited, etc.)

Implicit data: information that is not provided intentionally but gathered from available data streams (e.g. search history, order history, clicked on, accounts interacted with, etc.)

The item data consists mainly of an item’s features. In YouTube’s case that would be a video’s metadata such as title and description. For Zillow, this could be a home’s Zip Code, City Region, Price, or Number of Bedrooms for instance.

Other data sources could be external data (for example, Netflix might add external item data features such as box office performance or critic reviews) or expert-generated data (Pandora’s Music Genome Project uses human input to apply values for each song in each of approximately 400 musical attributes).

A key insight here is that obviously, having more data about your users will inevitably lead to better model results (if applied correctly), however, as Airbnb shows in their 3-part journey to building a Ranking Model for Airbnb Experiences you can already achieve quite a lot with lesser data: the team at Airbnb already improved bookings by +13% with just 500 experiences and 50k training data size.

“The main take-away is: Don’t wait until you have big data, you can do quite a bit with small data to help grow and improve your business.

Algorithms

Often, we associate Recommender Systems with just collaborative filtering. That’s fair, as in the past this has been the go-to method for a lot of the companies that have deployed successful systems in practice. Amazon was probably the first company to leverage item-to-item collaborative filtering. When they first released the inner workings of their method in a paper in 2003, the system had already been in use for six years.

Then, in 2006 Netflix followed suit with its famous Netflix Price Challenge which offered $1 million to whoever improved the accuracy of their existing system called Cinematch by 10%. Collaborative filtering was also a part of the early Recommender Systems at Spotify and YouTube. LinkedIn even developed a horizontal collaborative filtering infrastructure, known as Browsemaps. This platform enables rapid development, deployment, and computation of collaborative filtering recommendations for almost any use case on LinkedIn.

If you want to know more about collaborative filtering, I would recommend checking out Section 16 of Andrew Ng’s Machine Learning course on Coursera where he goes deeper into the math behind it.

Now, I would like to take a step back and generalize the concept of a Recommender System. While many companies used to rely on collaborative filtering, today there are a lot of other different algorithms at play that either complement or even replaced the collaborative filtering approach. Netflix went through this change when they shifted from a DVD shipping to a streaming business. As described in one of their papers:

“We indeed relied on such an algorithm heavily when our main business was shipping DVDs by mail, partly because in that context, a star rating was the main feedback that we received that a member had actually watched the video. […] But the days when stars and DVDs were the focus of recommendations at Netflix have long passed. […] Now, our recommender system consists of a variety of algorithms that collectively define the Netflix experience, most of which come together on the Netflix homepage.”

If we zoom out a little bit and look at Recommender Systems more broadly we find that they essentially consist of two parts:

  1. Candidate Generation
  2. Ranking

I am going to use YouTube’s Recommender System as an example below as they provided a good visualization, but that very same concept is applied by Instagram for recommendations in “Instagram Explore”, by Uber Eats in their Dish and Restaurant Recommender System, by Netflix for their movie recommendations and probably many other companies.

2-stage Recommender System (inspired by YouTube)

According to Netflix, the goal of Recommender Systems is to present a number of attractive items for a person to choose from. This is usually accomplished by selecting some items (candidate generation) and sorting them (ranking) in the order of expected enjoyment (or utility).

Let’s further investigate the two stages:

Candidate Generation

In this stage, we want to source the relevant candidates that could be eligible to show to our users. Here, we are working with the whole catalog of items so it can be quite large (YouTube and Instagram are great examples here). The key to doing this is entity embeddings. What are entity embeddings?

An entity embedding is a mathematical vector representation of an entity such that its dimensions might represent certain properties. Twitter has a great example of this in a blog post about Embeddings@Twitter: say we have two NBA players (Stephen Curry and LeBron James) and two musicians (Kendrick Lamar and Bruno Mars). We expect the distance between the embeddings of the NBA players to be smaller than the distance between the embeddings of a player and a musician. We can calculate the distance between two embeddings using the formula for Euclidean distance.

How do we come up with these embeddings?

Well, one way to do this would be collaborative filtering. We have our items and our users. If we put them in a matrix (for the example of Spotify) it could look like this:

After applying the matrix factorization algorithm, we end up with user vectors and song vectors. To find out which users’ tastes are most similar to another’s, collaborative filtering compares one users’ vector with all of the other users’ vectors, ultimately spitting out which users are the closest matches. The same goes for the Y vector, songs: you can compare a single song’s vector with all the others, and find out which songs are most similar to the one in question.

Another way to do this takes inspiration from applications in the domain of Natural Language Processing. Researchers generalized the word2vec algorithm, developed by Google in the early 2010s to all entities appearing in a similar context. In word2vec, the networks are trained by directly taking into account the word order and their co-occurrence, based on the assumption that words frequently appearing together in the sentences also share more statistical dependence. As Airbnb describes, in their blog post about creating Listing Embeddings:

More recently, the concept of embeddings has been extended beyond word representations to other applications outside of NLP domain. Researchers from the Web Search, E-commerce and Marketplace domains have realized that just like one can train word embeddings by treating a sequence of words in a sentence as context, the same can be done for training embeddings of user actions by treating sequence of user actions as context. Examples include learning representations of items that were clicked or purchased or queries and ads that were clicked. These embeddings have subsequently been leveraged for a variety of recommendations on the Web.

Apart from Airbnb, this concept is used by Instagram (IG2Vec) to learn account embeddings, by YouTube to learn video embeddings and by Zillow to learn categorical embeddings.

Another, more novel approach to this is called Graph Learning and it is used by Uber Eats for their dish and restaurant embeddings. They represent each of their dishes and restaurant in a separate graph and apply the GraphSAGE algorithm to obtain the representations (embeddings) of the respective nodes.

And last but not least, we can also learn an embedding as part of the neural network for our target task. This approach gets you an embedding well customized for your particular system, but may take longer than training the embedding separately. The Keras Embedding Layer would be one way to achieve this. Google covers this well as part of their Machine Learning Crash Course.

Once we have this vectorial representation of our items we can simply use Nearest Neighbour Search to find our potential candidates.
Instagram, for example, defines a couple of seed accounts (accounts that people have interacted with in the past) and uses their IG2Vec account embeddings to find similar accounts that are like those. Based on these accounts, they are able to find the media that these accounts posted or engaged with. By doing that, they are able to filter billions of media items down to a couple thousand and then sample 500 candidates from the pool and send those candidates downstream to the ranking stage.

This phase can also be guided by business rules or just user input (the more information we have the more specific we can be). As Uber Eats mentions in one of their blog posts, for instance, pre-filtering can be based on factors such as geographical location.

So, to summarize:

In the candidate generation (or sourcing) phase, we filter our whole content catalog for a smaller subset of items that our users might be interested in. To do this we need to map our items into a mathematical representation called embeddings so we can use a similarity function to find the most similar items in space. There are several ways to achieve this. Three of them being collaborative filtering, word2vec for entities, and graph learning.

Ranking

Let’s loop back to the case of Instagram. After the candidate generation stage, we have about 500 media items that are potentially relevant and that we could show to a user in their “Explore” feed.
But which ones are going to be the most relevant?

Because, after all, there are only 25 spots on the first page of the “Explore” section. And if the first items suck, the user is not going to be impressed nor intrigued to keep browsing. Netflix’s and Amazon PrimeVideo’s web interface shows only the top 6 recommendations on the first page associated with each title in its catalog. Spotify’s Discover Weekly Playlist contains only 30 songs.
Also, all of this is subject to the users’ device. Smartphones, of course, allowing for less space for relevant recommendations than a web browser.

“There are many ways one could construct a ranking function ranging from simple scoring methods, to pairwise preferences, to optimization over the entire ranking. If we were to formulate this as a Machine Learning problem, we could select positive and negative examples from our historical data and let a Machine Learning algorithm learn the weights that optimize our goal. This family of Machine Learning problems is known as “Learning to rank” and is central to application scenarios such as search engines or ad targeting. In the ranking stage, we are not aiming for our items to have a global notion of relevance, but rather look for ways of optimizing a personalized model” (Extract from Netflix Blog Post).

To accomplish this, Instagram uses a three-stage ranking infrastructure to help balance the trade-offs between ranking relevance and computation efficiency. In the case of Uber Eats, their personalized ranking system is “a fully-fledged ML model that ranks the pre-filtered dish and restaurant candidates based on additional contextual information, such as the day, time, and current location of the user when they open the Uber Eats app”. In general, the level of complexity for your model really depends on the size of your feature space. Many supervised classification methods can be used for ranking. Typical choices include Logistic Regression, Support Vector Machines, Neural Networks, or Decision Tree-based methods such as Gradient Boosted Decision Trees (GBDT). On the other hand, a great number of algorithms specifically designed for learning to rank have appeared in recent years such as RankSVM or RankBoost.

To summarise:

After selecting initial candidates for our recommendations, in the ranking stage, we need to design a ranking function that ranks items by their relevance. This can be formulated as a Machine Learning problem, and the goal here is to optimize a personalized model for each user. This step is important because in most interfaces we have limited space to recommend items so we need to make the best use of that space by putting the most relevant items at the very top.

Baseline

As for every Machine Learning algorithm, we need a good baseline to measure the improvement of any change. A good baseline to start with is just to use the most popular items in the catalog, as described by Amazon:

“In the recommendations world, there’s a cardinal rule. If I know nothing about you, then the best things to recommend to you are the most popular things in the world.”

However, if you don’t even know what is most popular, because you just launched a new product or new items — as was the case with Airbnb Experiences — you can just randomly re-rank the item collection daily until you have gathered enough data for your first model.


That’s a wrap for Part 1 of this series. There are a couple of points I wanted to emphasize in this article:

  • Recommender Systems are the most valuable application of Machine Learning as they are able to create a Virtuous Feedback Loop: the more people use a company’s Recommender System, the more valuable they become and the more valuable they become, the more people use them. Once you enter that Loop, the Sky is the Limit.
  • The right Problem Formulation is key.
  • In the Netflix Price Challenge, teams tried to build models that predict a users’ rating for a given movie. In the “real world”, companies use much more sophisticated data inputs which can be classified into two categories: Explicit and Implicit Data.
  • In today’s world, Recommender Systems rely on much more than just Collaborative Filtering.

In the Second Part I will cover:

  • Evaluation Metrics
  • User Interface
  • Cold-start Problem
  • Exploration vs. Exploitation

Resources

Airbnb — Listing Embeddings in Search Ranking

Airbnb — Machine Learning-Powered Search Ranking of Airbnb Experiences

Amazon — Amazon.com Recommendations Item-to-Item Collaborative Filtering

Amazon — The history of Amazon’s recommendation algorithm

Instagram — Powered by AI: Instagram’s Explore recommender system

LinkedIn — The Browsemaps: Collaborative Filtering at LinkedIn

Netflix — Netflix Recommendations: Beyond the 5 stars (Part 1)

Netflix — Netflix Recommendations: Beyond the 5 stars (Part 2)

Netflix — The Netflix Recommender System: Algorithms, Business Value, and Innovation

Netflix — Learning a Personalized Homepage

Pandora — Pandora’s Music Recommender

Spotify — Discover Weekly: How Does Spotify Know You So Well?

Spotify — For Your Ears Only: Personalizing Spotify Home with Machine Learning

Spotify — From Idea to Execution: Spotify’s Discover Weekly

Twitter — Embeddings@Twitter

Uber Eats — Food Discovery with Uber Eats: Recommending for the Marketplace

Uber Eats — Food Discovery with Uber Eats: Using Graph Learning to Power Recommendations

YouTube — The YouTube Video Recommendation System

YouTube — Collaborative Deep Learning for Recommender Systems

YouTube — Deep Neural Networks for YouTube Recommendations

Zillow — Home Embeddings for Similar Home Recommendations

Andrew Ng’s Machine Learning Course (Recommender Systems)

Google’s Machine Learning Crash Course — Embeddings

]]>
<![CDATA[Machine Learning System Design]]>https://rmeinl.com/machine-learning-system-design/6085751174348916f1b0899bMon, 02 Mar 2020 21:40:00 GMTRead this post on Medium.

Facebook Field Guide to Machine Learning

While preparing for job interviews I found some great resources on Machine Learning System designs from Facebook, Twitter, Google, Airbnb, Uber, Instagram, Netflix, AWS and Spotify.

I find this to be a fascinating topic because it’s something not often covered in online courses.

Twitter

Instagram

Facebook

Uber Eats

Uber

Airbnb

Airbnb Experiences

Linkedin

Google

Netflix

Spotify


In addition, here are some resources on a more general process. Starting with the book Data Science for Business which explains the CRISP-DM (Cross Industry Standard Process for Data Mining).

The process involves six stages:

  1. Business Understanding
  2. Data Understanding
  3. Data Preparation
  4. Modelling
  5. Evaluation
  6. Deployment

Here is a more high-level breakdown on how to apply CRISP-DM on AWS.

Facebook also created a video series where they go into depth in how they structure Machine Learning Projects with the Facebook Field Guide to Machine Learning.

]]>
<![CDATA[The World Doesn't Care - Lessons From Startup Failure]]>https://rmeinl.com/the-world-doesnt-care-lessons-from-startup-failure/6085751174348916f1b0899cMon, 16 Dec 2019 21:41:00 GMTThe other often overlooked and not-so-glamorous side of the founder’s story.

Read this article on Medium.

Photo by Sebastian Unrau on Unsplash

Everyone loves the good old startup story of the young college dropout who started from his dorm room, built an amazing product, assembled a team of equally smart people, worked incredibly hard and went on to change the world. In my opinion the ideology of starting your own startup has been completely overhyped in the recent years (maybe even longer but that’s as far as I can judge) and young people start companies just for the sake of being a part of this story. I thought it’d be interesting to contribute my story which often remains untold because we all shy away from admitting to failure.

My name is Rico, I am a 22 y/o entrepreneur and the technical co-founder of a FashionTech startup which we’ve recently put on hold. Having completed my first attempt of founding a startup I wanted to share a couple of painful lessons that I took away from this experience.

Before I get to my real list, let me quickly explain why I included my age in the previous sentence. This is actually the first fallacy I fell for: thinking you’re awesome because you went on a leave of absence from college to co-found a startup at age 21.
Breaking news: the world doesn’t care.
Investors don’t care. And most importantly, your customers don’t care. Don’t follow the hype stories of Zuckerberg, Gates and Jobs and think it gives you an advantage to start something while you’re young. It’s nice to have an idea and be able to start something while you’re young because you usually don’t have that much to lose and no family to take care of but it shouldn’t be the reason in the first place.

Now, to the actual learnings. Before you start reading this, I want to point out that I made 1 attempt to start a company so far and it didn’t succeed. This article is about what I learned not to do.
But, please take my advice with a grain of salt.
If you want to learn how to do it right, check out my blog. It only contains books recommended or written by billionaire founders.


As a quick background: I don’t like to go shopping and also don’t want to spend much of my time going to the mall. Still, I like to dress well. Online shopping is even worse because I have really long legs and nothing I ever ordered fit me well. My co-founder and I bonded through that problem when he had started working as a salesperson in a retail store in Hamburg to understand the customer needs in this segment. We joined forces in November 2018 and built an algorithm that allows people to measure themselves with their phone to find perfect fitting clothes. We started with only jeans to have focus and because so many people struggle to find a pair that fits well. In April 2019 we moved to Amsterdam and started working with two jeans brands and tried to sell the technology as a plugin to the online retailers (we talked with ASOS, Levi’s, Zappos, Nordstrom, AboutYou). In August 2019 we moved to Los Angeles and pivoted into B2C (all the major jeans brands are located there). Recently we put the company on hold.


#1 Start with a Problem in Your Own Life

We were building a product that we ourselves were the customer for. — Steve Jobs

This is probably the most painful but also the most valuable lesson of all. Especially because I had seen the video about Steve Jobs saying that a couple times. But there’s quite a difference between consuming information and then acting on it. We didn’t act on it. When we built the body measurement technology we decided to start with women. There were many reasons for that decision and I won’t go into depth here but what we painfully learned was that it is really hard to create a great consumer product. To get all the tiny details right, you’ll have a much better starting point when you’re building something for yourself.

To frame the problem we were trying to solve we used the “Jobs Theory” framework described by Clayton Christensen in his book “Competing Against Luck”: What job do people hire our product for?
Though, we failed to apply this framework to our solution, aka how our product helps people accomplish the jobs they hired it for. We realized this later on when we were already in L.A. and pivoted to men and couldn’t imagine how our product would fit into our own lives.

The learning here is:
Start with a problem in your own life and build a solution that you would use yourself.

#2 Stay Relentlessly Focused on the End Results with the Highest Impact on Your Business Strategy

Setting measurable goals is great. Setting the right goals is hard. There is a reason why executives and company founders get paid so much. The hardest job in the world is deciding and prioritizing what to work on. And if you work on the wrong things, it doesn’t matter how hard you work. You’re probably not gonna get where you want to go.

We learned this the hard way, when we spent three months trying to get to +-1% measurement inaccuracy to impress the big retailers (so we thought) and convince them to work with us. Our assumption was that our product needs to work really well to succeed.
In our field that meant, being able to recommend people jeans that fit perfectly. In order for the jeans to fit perfectly, it needs to sit well at the hip, waist, upper thigh, mid thigh, calf and ankle. That’s one reason why jeans are amongst the hardest products to fit (and therefore our reasoning: if we solve this, everything else will be like a walk in the park). So, even though it is true that all the body parts I mentioned above need to be measured with +-1% inaccuracy for the jeans to fit perfectly, there’s a relative degree of importance. We later found out that the hip and waist for example have the highest impact on the end result.

Taking it one step further, the +-1% inaccuracy was a terrible metric to focus on when trying to start a business. The online retailers don’t care about the measurement inaccuracy. They care about products that drive conversion and reduce costs associated with returns. Customers don’t care about the measurement inaccuracy. They don’t even care about the technology in general. They care about saving time and effort and eventually if the jeans makes them look and feel good.

It’s so easy to feel good about yourself when you work super hard and make progress towards achieving goals but none of that matters if those goals are not aligned with your business strategy. I feel like this one is especially important for tech people (such as myself) who love to build cool things. The stuff that matters most for the business is not always the most fun.

The learning here is:
Don’t get lost in the trenches and focus on the stuff that has the highest impact on the results you are trying to achieve as a business.

I’ll throw in another one here: Don’t build something that doesn’t have short-term value because “we’ll definitely need it in the future”.
I once cleaned up the whole code base for our web app and automatized one of our internal processes, just because I had some time on hand and thought that automation would definitely be useful and save time in the future. I was wrong. Not long after, the process changed and now it was super hard to add to it and I ended up wasting a lot of time reverting the automation I just implemented a week ago.

#3 It’s not the money. It’s the f****** money.

This is what Bill Campbell told Ben Horowitz when he was planning to take his company Loudcloud public (from “The Hard Thing About Hard Things”).

I thought this quote accurately captures the relationship you should entertain with money when starting a company. And by that I mean, how are you going to make money? As a tech person it’s easy to think that if you build something cool, you will somehow find a way for people to pay you for it.
But nowadays, especially in the era of mobile it’s not as easy as “just put ads on it”.

We made the mistake of thinking that our product has the potential to be so useful for people (save time and effort) and the online retailers (reduce online returns, which are at 60% for jeans) that we don’t need a business model but rather investors will fund us and we’ll figure out how to make money later. Needless to say, that was a bad strategy. During YC Startup School, Marc Andreessen once said the best way to raise money is to create a great business and then tell investors about it. Not the other way around.

The learning here is:
You’re running a business, not a research lab. If you don’t focus on the money, your business will die.

#4 Be careful with the latest research

I think this one heavily depends on the field you’re in and I don’t know if this is the case in general so I’m happy to hear about your experience. We started with three co-founders and my initial role was not the technology part. When we became two co-founders and I was in charge of getting to +-1% measurement inaccuracy my approach was to contact different companies that were already working on this to see if we could buy the technology.
None of them had something that was good enough so we had to move ahead and build it ourselves.
I locked myself in my room during all of November and December and basically read every research paper that was ever written in this field. That took a lot of time and effort because the barrier you need to overcome until you fully understand all these deeply technical papers is extremely steep. Not a very good idea, when you’re jumping off of a cliff trying to assemble a plane on the way down (Reid Hoffman).

In addition to that almost all of the papers are not industry-ready, which means they are presented nicely but have underlying flaws in terms of performance, they only work for specific edge cases or with lab data. My advice is to always connect with the authors and ask them to help you understand the algorithms and ask about the limits of the paper beforehand. Another thing I learned is that many papers are oversold. If you think about it, it makes sense because as a researcher you have to show process in your work to keep the funding going. It’s just good to be aware of that when you are trying to assess whether any of the ideas from the papers could be used in practice. In theory it looks amazing but in practice the performance is heavily limited.

The learning here is:
Unless you’re a PhD and know about what’s really possible in your field, be careful with trying to reimplement the latest research papers for your industry application.

#5 Reach out to Experts

Before we started our journey, we came across the book “Measure What Matters” by John Doerr, a heavily recommended book on OKRs (Objectives and Key Results), the management methodology from Intel which was implemented by Google back in the day and is still actively used.
Naturally we were intrigued to make this our standard methodology to set our monthly goals and how we’d get there. The book does mention the downside that it’s quite hard to implement and we struggled quite a bit with it. It just didn’t feel like it was dynamic enough because in a startup things change almost every day.
After 9 months I was tired of it and shot an email to some of the people interviewed in the book. Among them were Atticus Tysen, the CIO from Intuit and Brett Kopf, CEO of Remind. They both got back to me and their response was simple and clear: the methodology is too heavy and too much process for an early stage startup. Would’ve been good to know that before we started using it.

The learning here is:
Always reach out to experts and ask for their advice before you start to implement something that requires as much time and effort like OKRs.


Wrapping Up

To sum up, here are my 5 Learnings:

  1. Start with a problem in your own life and build a solution that you would use yourself.
  2. Don’t get lost in the trenches and focus on the stuff that has the highest impact on the results you are trying to achieve as a business.
  3. You’re running a business, not a research lab. If you don’t focus on the money, your business will die.
  4. Unless you’re a PhD and know about what’s really possible in your field, be careful with trying to reimplement the latest research papers for your industry application.
  5. Always reach out to experts and ask for their advice before you start to implement something that requires as much time and effort like OKRs.

Lastly, there are a lot of idealized founders’ stories out there but no one really talks about the other side:

  • The world doesn’t care how old you are (see above).
  • The world doesn’t care if it’s new years or your birthday. We were up until 4am in the night before my birthday, slept 2 hours and then went to a customer presentation in the morning. We worked all throughout new years eve.
  • The world doesn’t care how many hours you put in. 8.30am — 2.00am, 6 days a week.
  • The world doesn’t care where and how much you sleep. When we were sleeping in an office in LA we had to sleep on the floor because mattresses would’ve raised too much attention and in order to not get caught we had a sleeping window between 2am and 7am which was when the cleaning people left and the first people came to work. In Amsterdam we slept in a hostel room together with 30 other people.

I didn’t include this so you feel sorry for us. There are people in this world who live in much worse conditions. I included this to point out that when people are thinking about starting companies because they celebrate the stories where Elon Musk slept in his office instead of getting an apartment or when he talks about putting in 120h weeks during crazy periods at Tesla, that they forget there is another side to this coin.
Just because you do this doesn’t mean you will be successful and if you’re not, the world doesn’t care about it.

Do the one thing the world does care about:
Build great products that solve a real need in people’s lives, make the world a better place and make sure to become profitable with doing that so you can build a sustainable business for the long-term.

]]>