• e - ISSN No : 2832-4277
IJRTTE Logo

INTERNATIONAL JOURNAL OF RECENT TRENDS IN TECHNOLOGY AND ENGINEERING (IJRTTE)

A Transformer-Based Hybrid Framework for Context-Aware Text Summarization in Low-Resource Languages

Varsha Negi
Assistant Professor, Department of Computer Science, Shyam Lal College Evening, Shahdara, Delhi University, India.
K Ruth Isabels
Associate Professor, Department of Mathematics, Saveetha Engineering College (Autonomous), India.

Keywords: Transformer models, hybrid framework, context-aware summarization, low-resource languages, abstractive text summarization, multilingual NLP, semantic attention, knowledge-enhanced summarization, domain adaptation, deep learning.

Abstract

The rapid advancement of neural text summarization has largely benefited resource-rich languages, leaving low-resource languages underrepresented due to data scarcity and linguistic complexity. This paper proposes a Transformer-based hybrid framework that effectively generates context-aware abstractive summaries for low-resource languages by integrating the strengths of pre-trained multilingual transformer models with domain-specific hybrid enhancements. The framework leverages semantic attention mechanisms, extractive cues, and knowledge-guided rules to improve content relevance, coherence, and fluency in summaries. To address the limitations of existing approaches such as lack of contextual depth, poor generalizability, and over-reliance on large-scale datasets the proposed system introduces an adaptive learning pipeline tailored to low-resource environments. Experimental evaluations across multiple low-resource language datasets demonstrate significant improvements in ROUGE, BLEU, and METEOR scores compared to state-of-the-art baselines, while also maintaining low computational overhead. Furthermore, qualitative analysis reveals enhanced interpretability and domain adaptability. This research contributes a scalable, efficient, and linguistically inclusive solution to the growing need for automated summarization in underserved linguistic communities.
Download Certificate
Details

References

  1. Bai, Y., Gao, Y., & Huang, H. (2021). Cross-lingual abstractive summarization with limited parallel resources. arXiv preprint arXiv:2105.13648. https://doi.org/10.48550/arXiv.2105.13648
  2. Hasan, T., Bhattacharjee, A., Ray, B., Ghosh, S., & Shalia, R. (2021). XL-Sum: Large-scale multilingual abstractive summarization for 44 languages. arXiv preprint arXiv:2106.13822. https://doi.org/10.48550/arXiv.2106.13822
  3. Dabre, R., Lim, T., Kumar, A., Nakazawa, T., & Bhattacharyya, P. (2021). IndicBART: A pre-trained model for Indic natural language generation. arXiv preprint arXiv:2109.02903. https://doi.org/10.48550/arXiv.2109.02903
  4. Wu, Z., Wang, J., Zhang, C., & Bao, Y. (2020). Lite transformer with long-short range attention. arXiv preprint arXiv:2004.11886. https://doi.org/10.48550/arXiv.2004.11886
  5. Masih, S., Hassan, M., Fahad, L. G., & Hassan, B. (2025). Transformer-based abstractive summarization of legal texts in low-resource languages. Electronics, 14(12), 2320. https://doi.org/10.3390/electronics14122320
  6. Munaf, M., Afzal, H., Iltaf, N., & Mahmood, K. (2023). Low-resource summarization using pre-trained language models. arXiv preprint arXiv:2310.02790. https://doi.org/10.48550/arXiv.2310.02790
  7. Karthick, G. (2025). Context-aware topic modeling and intelligent text extraction using transformer-based architectures. SSRN. https://doi.org/10.2139/ssrn.4587391
  8. Choudhary, M., Yadav, S., & Prasad, R. (2025). Abstractive summarization of low resourced Nepali language using mBART and mT5. In Proceedings of the ChIPSAL Workshop 2025. https://aclanthology.org/2025.chipsal-1.12/
  9. Abbas, Z., Ahmad, W., & Anwar, N. (2023). Hybrid approach for automatic text summarization for low-resource languages. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(4), 1–17. https://doi.org/10.1145/3743677
  10. Zhao, X., Li, Y., & Sun, Q. (2023). Context-aware transformer pre-training for answer sentence selection. In Proceedings of the 61st Annual Meeting of the ACL (pp. 340–350). https://aclanthology.org/2023.acl-short.40/
  11. Kumar, V., & Bhardwaj, A. (2025). User query-based multilingual abstractive text summarization for low-resource Indian languages. Ain Shams Engineering Journal. https://doi.org/10.1016/j.asej.2025.102684
  12. Rathi, M., & Chandra, S. (2023). Multi-language transfer learning for low-resource legal case summarization. Artificial Intelligence and Law, 31(1), 45–62. https://doi.org/10.1007/s10506-023-09373-8
  13. Ghanem, F. A., Padma, M. C., Abdulwahab, H. M., & AlKhattab, R. (2025). Deep learning-based short text summarization: An integrated BERT and transformer encoder-decoder approach. Computation, 13(4), 96. https://doi.org/10.3390/computation13040096
  14. Cruz, J. C. B., & Aji, A. F. (2025). Extracting general-use transformers for low-resource languages via knowledge distillation. arXiv preprint arXiv:2501.12660. https://doi.org/10.48550/arXiv.2501.12660
  15. Nisha, R., & Prasad, A. (2025). LATS: Low resource abstractive text summarization. Expert Systems with Applications, 231, 119888. https://doi.org/10.1016/j.eswa.2024.119888
  16. Jiang, Y., Lin, B., & Qiu, X. (2024). Core context-aware transformers for long context language understanding. arXiv preprint arXiv:2412.12465. https://doi.org/10.48550/arXiv.2412.12465
  17. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008.
  18. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., ... & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140), 1–67. http://jmlr.org/papers/v21/20-074.html
  19. Narang, S., Lee, K., Chowdhery, A., & Devlin, J. (2024). T5X: Scaling up the T5 transformer. Google Research Blog. https://ai.googleblog.com/2024/01/t5x-scaling-up.html
  20. Xu, Y., Zhang, S., Yang, Y., & Zhang, M. (2023). Efficient transformer for low-resource summarization using structured compression. Proceedings of the AAAI Conference on Artificial Intelligence, 37(10), 11593–11601. https://doi.org/10.1609/aaai.v37i10.26654