different output

[sauravsinhaa]

when i ran textrank on jupyter there is difference in output

my querry=QueryExtractor("skydiving in dubai")
it is detect by jupyter ['dubai'.'skydiving]

it is detect by vs code ['dubai']

[Ankush-Chander]

Hi Saurav
Please share the complete code snippet for us to be able to replicate this issue.

Thanks

[sauravsinhaa]

import pytextrank
import googlemaps
import truecase
import spacy
from flair.data import Sentence
from flair.models import SequenceTagger
tagger = SequenceTagger.load('ner')
nlp_ner = spacy.load("QueryExtractor/MODEL/PERSONA")
nlp_spacy = spacy.load('en_core_web_sm')
nlp_spacy.add_pipe("textrank")
gmaps = googlemaps.Client(key='A')
nlp_date_custom = spacy.load("QueryExtractor/MODEL/dates1")
def location_recognition(item):
#item = truecase.get_true_case(item)
sentence = Sentence(item)
tagger.predict(sentence)
locations_spacy = [entity.text for entity in sentence.get_spans(
'ner') if 'LOC' in entity.tag]
# location=",".join(locations_spacy)
return locations_spacy
def date_recognition(item):
item=item.lower()
date_spacy = nlp_spacy(item)
dates_spacy = [ent.text for ent in date_spacy.ents if ent.label_ == "DATE"]
if dates_spacy:
return dates_spacy
else:
date_custom = nlp_date_custom(item)
dates_custom = [
ent.text for ent in date_custom.ents if ent.label_ == "DATE"]
return dates_custom
def persona_recognition(item):
doc_persona = nlp_ner(item)
persona_labels = [
ent.text for ent in doc_persona.ents if ent.label_ == "PERSONA"]
return persona_labels
def getCountry(recognized_locations):
try:
location = recognized_locations[0]
geocode_result = gmaps.geocode(location)
if geocode_result:
address_components = geocode_result[0]["address_components"]
for component in address_components:
if "country" in component["types"]:
return component["long_name"]
except Exception as e:
pass
return None
def contextFromQuery(textQuery):
doc = nlp_spacy(textQuery)
entity_phrases = [phrase.text for phrase in doc._.phrases[:10]]
print(entity_phrases)
recognized_locations = []
recognized_date = []
recognized_persona = []
persona = persona_recognition(textQuery)
recognized_persona.extend(persona)
date = date_recognition(textQuery)
recognized_date.extend(date)
locations = location_recognition(textQuery)
recognized_locations.extend(locations)
country = getCountry(recognized_locations)
travel_items = [
item for item in entity_phrases if item not in recognized_date]
return recognized_locations, recognized_date, recognized_persona, country, travel_items
def QueryExtractor(textQuery):
textQuery = truecase.get_true_case(textQuery)
recognized_locations, recognized_date, recognized_persona, country, travel_items = contextFromQuery(textQuery)
mydict = {}
mydict['locationItems'] = recognized_locations
mydict['dateItems'] = recognized_date
mydict['personaItems'] = recognized_persona
mydict['travelItems'] = travel_items
mydict['country'] = country
return mydict

[sauravsinhaa]

it is working fine on jupyter but not on vs code

[Ankush-Chander]

Hi Saurav,
It would be great if you can

1. format the code and make it easy to understand.
2. Share error logs or actual versus expected output
3. Versions of python, spacy, pytextrank as being used in vscode vs jupyter.

[ceteri]

Hi @sauravsinhaa,

It really helps to have the code formatted, otherwise we will have difficulty understanding about the indents, Python control flow, etc., in your code.

A quick way to do that is simply to place three back-tick characters "```" in a line before the start of the code that you've copy/pasted, and also in a line after the end of the code.

As another option, you can copy/paste into a "gist" at https://gist.github.com/ and then share the link in your GitHub issue.

[ceteri]

it is working fine on jupyter but not on vs code

Ah-ha! @sauravsinhaa, this is probably the most important point :)

One thing which is rather obscure and tricky about Jupyter is that when you run a Jupyter notebook locally from a command line, the Python environment gets determined by your shell environment variables, not by whatever Python venv or other virtualization you're currently working in.

I say that this is "obscure" because one of the lead developers for spaCy and I worked for several days to troubleshoot an issue, only to realize that Jupyter was running an entirely different environment.

If you are using venv and launch Jupyter directly from venv/bin then it will at least determine the versions of Python and Jupyter being run.

Otherwise, your shell will determine which Python executable gets run, not your locally specified one. Then your other Python library dependencies may all change suddenly.

FWIW, there isn't any code within pytextrank that runs differently in Jupyter vs. not in Jupyter. However, we have seen this problem several times before and the issues have always been that the environment used by Jupyter was quite different than what a developer expected.

BTW, to @Ankush-Chander's point, it really helps to install and use watermark when running in a Jupyter notebook, to analyze

• operating system version and compiler settings
• Python version
• Jupyter/IPython version
• versions of install libraries

[sauravsinhaa]

Yess it is the issues of environment