California Investment Network


Recent Blog


Pitching Help Desk


Testimonials

"For those of you that are asking yourself whether this site is real, the answer is yes. My first thought was that I would put my proposal on the site and it would be sent for review, and at this point someone from within the Dealflow Investment Network office would contact me as an investor so I would be more likely to pay the $249 fee. I received 8 responses from investors overnight and 2 more since then. Thanks Dealflow Investment Network."
David Kriedeman - Chris Christopherson Inc

 BLOG >> Recent

Startup Hypothesis Testing [Bayesian Inference
Posted on March 2, 2017 @ 06:44:00 AM by Paul Meagher

In my last lean startup blog on measurement, I talked about using a Minimal Viable Product (MVP) to test hypothesis derived from leap of faith assumptions contained in the startup vision.

In the case of Joe's lemonade stand (see previous blog), the first leap of faith was that the customer would buy the lemonade. Customers purchased the lemonade but not in the amounts he was looking for (10 customers). Joe then tested the price people would pay for it starting off at a premium price of $1.50 a glass. He measured sales volume at that price and another price ($1.00 a cup) and concluded that it was better to sell at the lower price because the volume more than compensated for the lower price. Joe is making progress towards operating a successful lemonade stand.

In this blog I want to look at the process of hypothesis testing in more detail and see how it maps onto some of the terms we have been using.

Let H be a class of hypothesis and h be a specific hypothesis.

Let M be a class of measurement outcomes and m be a specific measurement outcome.

We can use Modus Ponens (latin for "the affirming mode") to draw conclusions about whether our hypothesis is true:

If H=h Then M=m
M=m
-------------
H=h

We can also use Modus Tollens (latin for "the denying mode") to draw conclusions about whether our hypothesis is true:

If H=h Then M=m 
Not M=m 
--------------
Not H=h

So far we are in the realm of formal logic and these two forms of inference are foundational in guiding automated forms of inference.

We can cross over into the realm of informal logic by using the P( ) operator around all our assertions, where P stands for "the probability of".

So Modus Ponens now looks like this:

P(If H=h Then M=m)
P(M=m)
------------------
P(H=h)

And Modus Tollens now looks like this:

P(If H=h Then M=m)
P(Not M=m)
------------------
P(Not H=h)

It is this form of Modus Ponens and Modus Tollens that we are dealing with when we test our startup assumptions. The application of scientific methods to startup hypothesis does not necessarily yield clear cut answers, but answers where one hypothesis might seem be better supported by the evidence than another hypothesis, without being able to completely rule out an alternative hypothesis.

In the case of the learning platform company Grockit (see previous blog), they were adding new peer-learning features to their learning platform and not seeing any effects on their metrics. They concluded that the learner only wanted peer-learning up to a point, then the learner wanted to engage in solo mode learning. A logical possibility was also that Grockit didn't zone in on the proper peer-learning approach yet. The alternative hypothesis is not completely ruled out by testing and measurement, but made sufficiently implausible that a pivot was deemed necessary.

We will be getting into the topic of pivoting in the next blog, but it is important to note here that deciding when to pivot or not is made difficult by the fact that the original and alternative hypothesis may each have merit making it difficult to decide what to do.

Recognizing that probabilities are involved can be helpful in deciding what decision making framework you want to use in your startup hypothesis testing. If P(H=h) is .6 perhaps that is enough certainty to go by in situations of irreducible uncertainty (you don't have the time or resources to achieve greater certainty).

You could examine formula-laden articles on sequential A/B testing and bayesian A/B testing to try to figure out when to stop collecting data and what to conclude (which I recommend reading), but I'm also interested in a more practical approach based on using informal logic to evaluate the probability of the premises P and the probability of the inference (i.e., P(if P Then C)) to arrive at a probability of the conclusion C of an argument.

P(If P Then C)
P(P)
---------------
P(C)

The evaluation of the premises and the inferences is based upon informal logic techniques appropriate to criticizing scientific arguments, combined with common sense, to assign probabilities to each premise. The evaluation of the premises and the inferences of the argument determines the evaluation you assign to the conclusion. Bayesian forms of informal logic may also involve assigning a prior probability to the conclusion so that the posterior probability of the conclusion can be evaluated.

P(If P Then C)
P(P)
P(C)
---------------
P(C)

Whether these probabilities are to be combined additively or multiplicatively to yield the posterior conclusion is worth thinking about, although multiplicative combination tends to used more often and to work better. Informal logic nowadays often involves creating a graphical representation of the argument. Below is how we might graphically express this Bayesian approach to evaluating arguments (where hypothesis testing is just one type of argument). The premises (e.g., the measurements and other assumptions) appear at the top with lines connecting them to the conclusion. The lines are your inferences (if P1 then C, if P2 then C). The prior probability of the conclusion C (based on previous knowledge) appears next to the premises as a separate contribution to the posterior conclusion probability C. The posterior probability of the conclusion at the bottom is what you get when you combine your prior probability of C and a likelihood estimate (the left side of the argument below).

The purpose of this blog was to dig a bit deeper into what startup hypothesis testing might involve from an formal and informal logic perspective. I am not a practicing logician and this is not a peer reviewed discussion so you may or may not find this a useful framework to use when approaching the problem of testing the leaps of faith that your startup vision implies.

Inspiration for this blog and the argument evaluation diagramming comes form my undergraduate mentor Wayne Grennan and his book Informal Logic (1997).

Ian Flemming in his excellent book Lean Logic (2016) has this to say about the relationship between informal and formal logic.

It sounds banal, but the syllogisms of formal logic are the building blocks of reasoning, which - in combination with a series of conditions, affirmed or denied in sequence and in parallel - can develop into a problem-solving capacity of great complexity, used as the logical structure on which artificial intelligence is based.

Informal logic is, of course, the junior partner in all this, since it depends on the reasoning of formal logic, and its mixing up of logic and content is exactly what you cannot do with formal logic. On the other hand, without content, logic has no purpose. Formal logic is the road, informal logic is the journey. ~ p. 165

Permalink 

 Archive 
 

Archive


 November 2023 [1]
 June 2023 [1]
 May 2023 [1]
 April 2023 [1]
 March 2023 [6]
 February 2023 [1]
 November 2022 [2]
 October 2022 [2]
 August 2022 [2]
 May 2022 [2]
 April 2022 [4]
 March 2022 [1]
 February 2022 [1]
 January 2022 [2]
 December 2021 [1]
 November 2021 [2]
 October 2021 [1]
 July 2021 [1]
 June 2021 [1]
 May 2021 [3]
 April 2021 [3]
 March 2021 [4]
 February 2021 [1]
 January 2021 [1]
 December 2020 [2]
 November 2020 [1]
 August 2020 [1]
 June 2020 [4]
 May 2020 [1]
 April 2020 [2]
 March 2020 [2]
 February 2020 [1]
 January 2020 [2]
 December 2019 [1]
 November 2019 [2]
 October 2019 [2]
 September 2019 [1]
 July 2019 [1]
 June 2019 [2]
 May 2019 [3]
 April 2019 [5]
 March 2019 [4]
 February 2019 [3]
 January 2019 [3]
 December 2018 [4]
 November 2018 [2]
 September 2018 [2]
 August 2018 [1]
 July 2018 [1]
 June 2018 [1]
 May 2018 [5]
 April 2018 [4]
 March 2018 [2]
 February 2018 [4]
 January 2018 [4]
 December 2017 [2]
 November 2017 [6]
 October 2017 [6]
 September 2017 [6]
 August 2017 [2]
 July 2017 [2]
 June 2017 [5]
 May 2017 [7]
 April 2017 [6]
 March 2017 [8]
 February 2017 [7]
 January 2017 [9]
 December 2016 [7]
 November 2016 [7]
 October 2016 [5]
 September 2016 [5]
 August 2016 [4]
 July 2016 [6]
 June 2016 [5]
 May 2016 [10]
 April 2016 [12]
 March 2016 [10]
 February 2016 [11]
 January 2016 [12]
 December 2015 [6]
 November 2015 [8]
 October 2015 [12]
 September 2015 [10]
 August 2015 [14]
 July 2015 [9]
 June 2015 [9]
 May 2015 [10]
 April 2015 [9]
 March 2015 [8]
 February 2015 [8]
 January 2015 [5]
 December 2014 [11]
 November 2014 [10]
 October 2014 [10]
 September 2014 [8]
 August 2014 [7]
 July 2014 [5]
 June 2014 [7]
 May 2014 [6]
 April 2014 [3]
 March 2014 [8]
 February 2014 [6]
 January 2014 [5]
 December 2013 [5]
 November 2013 [3]
 October 2013 [4]
 September 2013 [11]
 August 2013 [4]
 July 2013 [8]
 June 2013 [10]
 May 2013 [14]
 April 2013 [12]
 March 2013 [11]
 February 2013 [19]
 January 2013 [20]
 December 2012 [5]
 November 2012 [1]
 October 2012 [3]
 September 2012 [1]
 August 2012 [1]
 July 2012 [1]
 June 2012 [2]


Categories


 Agriculture [77]
 Bayesian Inference [14]
 Books [18]
 Business Models [24]
 Causal Inference [2]
 Creativity [7]
 Decision Making [17]
 Decision Trees [8]
 Definitions [1]
 Design [38]
 Eco-Green [4]
 Economics [14]
 Education [10]
 Energy [0]
 Entrepreneurship [74]
 Events [7]
 Farming [21]
 Finance [30]
 Future [15]
 Growth [19]
 Investing [25]
 Lean Startup [10]
 Leisure [5]
 Lens Model [9]
 Making [1]
 Management [12]
 Motivation [3]
 Nature [22]
 Patents & Trademarks [1]
 Permaculture [36]
 Psychology [2]
 Real Estate [5]
 Robots [1]
 Selling [12]
 Site News [17]
 Startups [12]
 Statistics [3]
 Systems Thinking [3]
 Trends [11]
 Useful Links [3]
 Valuation [1]
 Venture Capital [5]
 Video [2]
 Writing [2]