Skip to content

[metadata] Remove metadata from empty enriched items#848

Open
animeshk08 wants to merge 3 commits intochaoss:mainfrom
animeshk08:metadate_fix
Open

[metadata] Remove metadata from empty enriched items#848
animeshk08 wants to merge 3 commits intochaoss:mainfrom
animeshk08:metadate_fix

Conversation

@animeshk08
Copy link
Copy Markdown
Contributor

@animeshk08 animeshk08 commented Apr 15, 2020

Fixes: #846
This commit updates the enricher for
github and gitlab to prevent addition of
metadata fields to empty enriched items

Signed-off-by: Animesh Kumar animuz111@gmail.com

@animeshk08
Copy link
Copy Markdown
Contributor Author

animeshk08 commented Apr 15, 2020 •
edited
Loading

Hi @valeriocos. I have checked the enrichers for all the backends. Even though most of the backend override the enrich_items method, they do not require this fix as there no category based checks.
In other words, in all the other backends except GitHub and GitLab the enrich items are never empty.

Please have a look at this PR :)

Also, I will update the tests and separate the commits tomorrow :)

@coveralls
Copy link
Copy Markdown

coveralls commented Apr 15, 2020 •
edited
Loading

Pull Request Test Coverage Report for Build 2135

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 219 unchanged lines in 4 files lost coverage.
  • Overall coverage increased (+0.06%) to 79.573%
Files with Coverage Reduction New Missed Lines %
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/github2.py 16 94.19%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/gitlab.py 20 87.8%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/github.py 38 75.93%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/enrich.py 145 73.6%
Totals Coverage Status
Change from base Build 2133: 0.06%
Covered Lines: 7904
Relevant Lines: 9933
💛 - Coveralls

@animeshk08
[metadata] Remove metadata from empty enriched items
719604f 
This commit updates the enricher to prevent
addition of metadata fields to empty enriched items.

Signed-off-by: Animesh Kumar <animuz111@gmail.com>
@animeshk08
[github] Remove metadata from non-enriched items
0574e55 
This commit prevents addition of repository label
and metadata filter raw to items which are not
to be enrihced. This results in creation of empty
items which are not uploaded to ElasticSearch.
Tests have been added accrodingly.

Signed-off-by: Animesh Kumar <animuz111@gmail.com>
@animeshk08
[gitlab] Remove metadata from non-enriched items
0036e06 
This commit prevents addition of repository label
and metadata filter raw to items which are not
to be enrihced. This results in creation of empty
items which are not uploaded to ElasticSearch.
Tests have been added accrodingly.

Signed-off-by: Animesh Kumar <animuz111@gmail.com>
@animeshk08
Copy link
Copy Markdown
Contributor Author

animeshk08 commented Apr 24, 2020

Hi, @valeriocos. Please have a look at this PR :)
SInce only GitHub and GitLab had such a category check of items the PR covers only these backends.
Do you think we should extend this check to other backends as well? This will serve two main purposes:

  • In case a raw item is not to be enriched at all.
  • In case fetching of a new category of item is integrated and work is being done on creating its enricher.
    WDYT?

@valeriocos
Copy link
Copy Markdown
Member

valeriocos commented Apr 24, 2020

Hi @animeshk08 , sorry for the late reply! ^^

We are testing it in the pre-production environment, and the initial tests are good.
It will be probably go into the release 0.2.41 (not the upcoming one, but the next one).

@animeshk08
Copy link
Copy Markdown
Contributor Author

animeshk08 commented Apr 24, 2020

Thank you for the quick reply @valeriocos. Let me know when there are any updates.
We can extend the discussion on #848 (comment) then :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[enrich] Update decorator to add metadeta only for non-empty items

3 participants

@animeshk08 @coveralls @valeriocos