How to Choose a Methodology for a Data Science or Machine Learning Dissertation (2026): Model Selection, Validation and Reproducibility

A machine learning methodology chapter is judged on a different failure mode than a traditional computer science systems chapter. The code can run, the accuracy can look impressive, and the chapter can still fail if the split between training and test data leaked, if the number reported is a single lucky run, or if a stronger baseline was never tried. This guide sets out the methodology decisions a data science or machine learning dissertation has to defend: how to split data without leakage, when cross-validation replaces a held-out test set, which evaluation metric actually answers your research question, and what a reproducibility package now has to contain.

How this differs from a general computer science methodology chapter

• A systems dissertation defends an architecture; an ML dissertation defends an evaluation protocol. The chapter has to earn trust in the number, not just the model.
• The single most common fatal flaw is data leakage: information from the test set influencing training, directly or through preprocessing. Kaufman, Rosset and Perlich’s widely cited taxonomy (2012, ACM Transactions on Knowledge Discovery from Data) is worth citing by name if your committee has seen a leakage failure before — most have.
• A single accuracy number from one run is not a result. Report variance across multiple random seeds or folds, not a point estimate presented as if it were exact.
• Your baseline decides whether your result means anything. A proposed model that beats a weak or absent baseline has not shown anything; state the baseline before the result.
• Reproducibility is no longer optional politeness. Committees increasingly expect the code, the exact environment, and the data version alongside the write-up, following the spirit of the machine learning reproducibility standards the field has converged on since 2020.

Framing the research question as a testable comparison

Before any model runs, the methodology chapter has to state a comparison a reader can evaluate: which model, on which data, measured by which metric, beats which baseline, by how much, and is the difference larger than what chance and randomness could produce. A vague framing — “apply deep learning to X” — gives the examiner nothing to check. A precise one — “does a gradient-boosted model outperform logistic regression on this imbalanced classification task, measured by the area under the precision-recall curve, by more than the variation seen across five random seeds” — gives the whole chapter its structure.

Splitting data without leakage

Scikit-learn’s own documentation states the core rule plainly: “learning the parameters of a prediction function and testing it on the same data is a methodological mistake.” The same warning extends to every preprocessing step — scaling, imputation, feature selection — which must be fitted on the training fold only and then applied, not refitted, to the validation or test data. A scaler fitted on the whole dataset before the split has already leaked information about the test set into training, even though no label was copied. If your data has a time dimension, groups (multiple rows per patient, per firm, per student), or duplicate near-identical records, a naive random split leaks structure too; use a time-based split, a group-aware split, or deduplication before splitting, and say explicitly in the chapter which one you used and why.

Cross-validation versus a single held-out test set

K-fold cross-validation trains on k-1 folds and validates on the remaining fold, repeating k times and averaging the score, which uses the data more efficiently than a fixed validation split and gives a spread, not just a mean. The standard practice this dissertation-level chapter should follow: use cross-validation on the training portion of the data to select and tune the model, and hold out a separate test set, touched exactly once, for the final reported number. Reporting a cross-validation score as if it were the test score, or tuning hyperparameters against the test set and then reporting that same test set’s score, both count as leakage even though no code error occurred — the mistake is procedural, not computational.

Choosing the evaluation metric your question actually needs

The metric is a methodological choice, not an afterthought pasted from a tutorial. For classification: accuracy is misleading whenever classes are imbalanced, so precision, recall, F1 and the area under the ROC or precision-recall curve should be reported alongside it, and the choice between ROC and precision-recall should be justified by how rare the positive class is. For regression: RMSE penalizes large errors more than MAE does, and R-squared alone hides whether the errors are systematically biased for some part of the range — a residual plot belongs in the chapter, not just the appendix. State the metric and the reason for it before showing a single result table, so the table cannot be read as having been chosen after the fact to flatter the model.

State this choice in the methodology chapter itself, before any results appear, and keep the same metric throughout the comparison between your baseline and your proposed model. Switching the headline metric between the baseline table and the proposed-model table — even when both numbers are technically correct — reads as having picked whichever metric flattered each model most, and it is one of the fastest ways to lose an examiner’s trust in an otherwise solid chapter.

Baselines and ablations

Every result needs a floor and, where the contribution is a specific component, a ceiling test. The floor is a simple, well-understood baseline: a majority-class predictor, a linear or logistic model, or the previous best published method on the same data, run under exactly the same split and metric as the proposed model. The ceiling test, an ablation, removes one component of your proposed method at a time and reports the metric drop, showing which part of the contribution is actually doing the work rather than crediting the whole pipeline for a gain that came from one piece of it.

Reporting variance, not a single lucky run

Neural network training in particular is sensitive to random initialization, data shuffling order, and even hardware nondeterminism. Report the mean and standard deviation (or a confidence interval) across a stated number of runs with different random seeds, not a single number from the run that happened to look best. The methodological standard the field has converged on since Pineau and colleagues’ 2020 paper in the Journal of Machine Learning Research, “Improving Reproducibility in Machine Learning Research,” asks authors to report exactly this: the number of runs, the variation across them, the hyperparameters searched, and the compute used to find them, so a reader can judge whether a reported improvement survives ordinary noise.

The reproducibility package your committee will expect

A methodology chapter is only as credible as the record behind it. At minimum, keep and reference: the exact code version (a commit hash or a tagged release, not “the code on my laptop”), a pinned environment file listing every library version, the random seeds used for every reported run, the exact train/validation/test split or the code that generates it deterministically, and a versioned copy or checksum of the dataset if it cannot be redistributed. The FAIR principles — that research outputs should be Findable, Accessible, Interoperable and Reusable, set out by Wilkinson and colleagues in 2016 in Scientific Data — are the standard your data and code management section should be able to point to, even when the underlying dataset itself has to stay restricted for privacy or licensing reasons.

Common mistakes in an ML methodology chapter

• Fitting preprocessing before the split. Any transformation learned on the full dataset before splitting leaks test information into training.
• Tuning hyperparameters against the test set. Use cross-validation on the training data for tuning; touch the test set once, at the end.
• Reporting accuracy alone on an imbalanced dataset. Add precision, recall and a curve-based metric appropriate to how rare the positive class is.
• No baseline, or a strawman baseline. A believable baseline is the only thing that gives a beaten baseline any meaning.
• One run, reported as the result. State the number of seeds or folds and report the spread, not just the best number you saw.

This chapter sits inside the wider document our step-by-step guide to writing a computer science dissertation maps out, which covers scope, version control and citation format but stops short of the ML-specific evaluation protocol this guide sets out. If your outcome variable is continuous versus a class label, our comparison of linear and logistic regression covers the classical decision your proposed model is likely being benchmarked against, and the general selection matrix in which statistical test should I use covers the significance-testing layer once your metric is chosen. Our field guide on research reproducibility and open-science practice covers the discipline-general version of the reproducibility package this chapter needs; the same evaluation-protocol discipline, applied to a very different kind of dissertation, is covered in our guide to writing the results chapter of an economics dissertation. For the write-up itself once the protocol is run, see writing a results chapter faster with AI.

Frequently asked questions

What is data leakage and why does it matter for a dissertation?

Data leakage happens when information from outside the training set, most often from the validation or test data, influences the model during training or tuning. It produces a reported score that looks strong but does not generalize, and it is the single most common reason an ML methodology chapter is sent back for revision. Fitting any preprocessing step on the full dataset before splitting is the most frequent source.

Should I use cross-validation or a single train-test split?

Use cross-validation on your training data to select and tune the model, since it uses the data more efficiently and reports a spread rather than a single number, and hold out a separate test set that is touched exactly once for the final reported result. Reporting the cross-validation score itself as the final test result is a common and avoidable error.

How many random seeds should I report results across?

There is no universal number, but one seed is not defensible for a dissertation-level claim. Report the mean and standard deviation, or a confidence interval, across at least three to five runs with different seeds, and state the exact number you used so a reader can judge whether your effect exceeds ordinary training noise.

Do I need a baseline if my method is genuinely novel?

Yes, always. A novel method still needs a simple baseline (a majority-class predictor, a linear model, or the best prior published result on the same data and split) so the reader has something to compare the novel result against. Without one, a reported improvement has no reference point.

What should my code and reproducibility appendix actually contain?

At minimum: the exact code version, a pinned list of library versions, the random seeds used, the code or exact procedure that generates your data split deterministically, and a versioned reference or checksum for the dataset if it cannot be redistributed. State this explicitly rather than assuming the examiner will ask for the code separately.

Is accuracy a good metric for an imbalanced classification dataset?

Rarely on its own. If one class is much rarer than the other, a model that always predicts the majority class can score high accuracy while being useless. Report precision, recall, F1 and a curve-based metric such as the area under the precision-recall curve, and justify the choice by how rare the positive class actually is in your data.