Motivation#
Recently, I carried out a data analysis project which studied the UK House Price Index, using data which is published by the HM Land Registry. For me, this was primarily a career exploration project. My aim was to experience the exploratory side of Data Science: exploring and analysing a dataset and using statistical techniques in order to test hypotheses.
I was drawn to this dataset because I wanted to avoid redoing the “classic” problems in data science which are pursued by aspiring practitioners. I wanted to be challenged, and so I sought out a real dataset published by a real organisation.
What I Learnt#
During my university studies, I studied a few courses which had components involving some form of data science or applied AI. However, the coursework in these modules was heavily structured. Therefore, this project introduced me to the level of decision making required in data analysis since when looking at a dataset that you’ve never seen before and have little context about, it can be difficult to know how to start and what is worth pursuing.
Doing a joint honours, my degree was split almost 50/50 between Computer Science and Maths. On the maths side, I elected for either pure or applied maths courses, choosing none of the available statistics-based courses. Therefore, my only exposure to statistics were from the aforementioned data science courses. This was sufficient for exploratory data analysis, however for statistical testing, I had to learn (informally) about regression analysis and variance testing.
This project gave me a wider appreciation for the precision demanded by data science. While documenting the analysis in the notebooks, I had to be very careful not to call results that haven’t been statistically tested “significant”, or make causal claims that were not supported by the dataset. It taught me to distinguish between what the data shows, what statistical analysis supports, and what can only be treated as a possible explanation.
Overall, the project gave me a better understanding of the decision-making and statistical reasoning involved in carrying out independent data analysis. The full analysis, including the methodology, statistical testing, and results is available in the project repository linked below.