Using the test set as part of an unsupervised pre-training step and then reporting the results on the same test set is not considered fair and can be seen as a form of overfitting to the test set. In machine learning, it is important to have separate datasets for training, validation, and testing to properly evaluate the performance of a model and ensure its generalizability.
The purpose of the test set is to assess the model's performance on unseen data that it has not been exposed to during training. If you include the test set as part of the pre-training process, the model can gain information about the specific samples in the test set, which can bias its performance evaluation.
Overfitting refers to a situation where a model performs well on the training data but fails to generalize to new, unseen data. By incorporating the test set into the pre-training step, the model might learn to exploit specific characteristics or patterns unique to the test set, leading to artificially inflated performance when evaluated on the same test set.
To avoid overfitting and obtain a fair evaluation, it is recommended to separate the dataset into three distinct parts: a training set, a validation set, and a test set. The training set is used to train the model, the validation set is used for tuning hyperparameters and model selection, and the test set is used for a final unbiased evaluation.