[Enrich] Add copy_raw_fields() method and refactor enrichers

[imnitishng]

This PR removes the following redundant code snippet from enrichers

for f in self.RAW_FIELDS_COPY:
    if f in item:
        eitem[f] = item[f]
    else:
        eitem[f] = None

and replaces with new method copy_raw_fields()
Based on the discussion here #851 (comment)

Signed-off-by: Nitish Gupta imnitish.ng@gmail.com

[coveralls]

Pull Request Test Coverage Report for Build 2174

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 369 unchanged lines in 32 files lost coverage.
• Overall coverage decreased (-0.2%) to 81.33%

---

Files with Coverage Reduction	New Missed Lines	%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/confluence.py	1	96.59%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/crates.py	1	80.26%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/githubql.py	1	98.61%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/puppetforge.py	1	96.94%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/supybot.py	1	96.08%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/bugzillarest.py	2	95.18%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/gitter.py	2	96.34%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/phabricator.py	2	67.89%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/redmine.py	2	97.47%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/telegram.py	2	96.15%

Totals
Change from base Build 2170:	-0.2%
Covered Lines:	7972
Relevant Lines:	9802

---

💛 - Coveralls

[valeriocos]

Thank you @imnitishng for the PR.

Can you add a test to https://github.com/chaoss/grimoirelab-elk/blob/master/tests/test_enrich.py to check that the method copy_raw_fields works fine

Probably some code could be added also here: https://github.com/chaoss/grimoirelab-elk/blob/master/tests/base.py#L197 to check that the default fields are copied to each enriched document.

The code could be as the one below:

        url = self.es_con + "/" + self.enrich_index + "/_search"
        response = enrich_backend.requests.get(url, verify=False).json()

        for hit in response['hits']['hits']:
            source = hit['_source']
            ...check that the default fields have been copied

WDYT?

[imnitishng]

Sure, will update the PR.

[valeriocos]

Thanks @imnitishng

[imnitishng]

Hi @valeriocos, I have added the tests here https://github.com/chaoss/grimoirelab-elk/blob/master/tests/test_enrich.py .
But I wonder if adding tests at https://github.com/chaoss/grimoirelab-elk/blob/master/tests/base.py#L197 might not be efficient, since in every enricher _test_raw_to_enrich() is called multiple times, causing the unnecessary overhead for checking raw fields for every test item multiple times for every backend.
I think adding a new test_raw_fields() in every backend would be a better option for a new PR.

WDYT?

[valeriocos]

Good idea @imnitishng !

Ok for the new test (the name could be test_copy_raw_fields, since we are testing that some fields are copied from the raw index to the enriched index).

I would add this new test in this PR (as a separated commit) to make sure that ELK is working properly after this modification. WDYT?

[imnitishng]

I have updated the tests @valeriocos, please have a look whenever you're free.

[valeriocos]

Thank you @imnitishng for the PR. I left a minor comment

[valeriocos]

this should be assertIsNone

[valeriocos]

LGTM, thanks @imnitishng