Custom Greek article crawler & classifier

Table of contents

• General info
• Technologies
• Pre-trained model
• Custom-trained model

General info

ArachneClassifier is a project for crawling, indexing, analyzing and classifying Greek articles.
Currently it crawls pages from a Greek news site, specifically, crime articles and classifies them based on the crime type. Crawling and classification is customizable.

Technologies

Requirements

install frozen-requirements from the main folder and requirements from the dash subfolder

Pre-trained model

To use the pre-trained model that classifies Greek article crime types install and set up elasticsearch and run:

python manage.py search_index --rebuild

to create the elasticsearch analyzers with which we will preprocess each article. Each article is tokenized, lowercased, stemmed and stop words are removed from it.

Use dash/app.py to open up a flask api where you paste a crime article and it classifies the type.

Custom-trained model

To crawl different pages go to:

. 
├── crawling
├── crawling
└── spiders

and edit the spider.

If you intent to change the structure of the data (fields) then edit the model in:

.
├── api 
└── article_models

To initiate the scraping run: python scrapy crawl newsbomb, the text is saved in a mongo db (djongo is used to communicate with the django model).
Then, index the djongo database with elasticsearch, run: python python manage.py search_index --rebuild
now that the data is indexed from elastic, go to:

.
├── nlp_classification
└── ML_classification

and uncomment the export_dataset_df() function. This gathers all the analyzed data from elastic and exports it to a dataframe.
The model is trained with SVM and exported.
Open dash/app.py to test the custom classification.