Text pre-processing

[yvkrishna]

As part of this issue, we can discuss and look out for various ways to pre-processes textual data

1. Removing Stop Words,
2. Removing special characters,
3. Convert Accented characters,
4. Lemmatization and stemming

[soma2000-lang]

@yvkrishna please assign me this

[rubyruins]

@yvkrishna, is this issue open? If so, I can work on it

[yvkrishna]

yes @rubyruins, You can work on this issue.

[rubyruins]

@yvkrishna, sure thing, I will start on this.

[rubyruins]

@yvkrishna what format should be used for the inputs? and does the user get a choice in selecting which functions to apply? Also, do you want the functions to be applied and saved separately or done serially on the same piece of text?

[yvkrishna]

@yvkrishna what format should be used for the inputs? and does the user get a choice in selecting which functions to apply? Also, do you want the functions to be applied and saved separately or done serially on the same piece of text?

It would be better to have a dictionary where the key would be the pre processing method and the value would be corresponding values required to implement that method.