In many text mining applications, the side-information contained within the text document will contribute to enhance the overall clustering process. The proposed algorithm performs clustering of data along with the side information, by combining classical partitioning algorithms with probabilistic models to boost the efficacy of the clustering approach. The clusters generated will be used as a training model to solve the classification problem. The proposed work will also make use of a similarity based ontology algorithm, by incorporating two shared word spaces, to perk up the clustering approach. This will boost the amount of knowledge gained from text documents by including ontology with side-information.