Early detection and intervention at the milder stages of cognitive impairment, e.g., Mild Cognitive Impairment (MCI), have been found to be crucial. Classic diagnostic models are usually built on unimodal data, such as EEG or speech signals only. However, unimodal systems have drawbacks: EEG methods provide neurophysiological relevance; it is sensitive to noise, and demanding hardware; speech methods, if facile, lack depth to capture subtle neurological disorders, especially in a wide range of people. In this paper, we present a new approach of multi-modal deep learning model which combines speech and EEG signals to overcome these issues and enhance diagnostic accuracy. By integrating linguistic features through which cognitive behaviours are described with neurophysiological traces from EEG, the model develops a broader account of cognitive function. We utilize a novel hybrid deep learning network comprising Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) modules, and attention mechanisms for extracting and aligning the temporal and inter-modality context. With this, the approach shows superior performance than the other state of the art unimodal approaches in terms of classification accuracy, robustness and generalization. In addition, the framework is non-invasive, real-time ready, and scalable and is applicable for integration in telehealth plat-forms and smart screening systems. It also allows for the continuous monitoring of cognitive function needed to follow the progression of the disease and the effects of treatment. To sum up, compared with previous unimodal models, our models have risen above their limitations and provide a clinically practical, scalable and accurate complementary solution for early detection of cognitive impairment with a multi-modal DL approach.