upmap-remapped.py erroring on loading existing upmaps

[carlilek]

I am experiencing an issue with this tool on Rocky 9.4, with Python 3.9.18 and Reef 18.2.4. I have the rados 2.0.0 module. When I run upmap-remapped.py, I get this:

# ./upmap-remapped.py
Error loading existing upmaps

Digging around a bit, it appears to be erroring specifically on line 162:
upmaps = json.loads(osd_dump_json)['pg_upmap_items']

If I print the exception, I get:
Error loading existing upmaps: Expecting value: line 1 column 2345 (char 2344)

Now, if I put the data from line 161:
osd_dump_json = output.decode('utf-8').strip()
into a file and run jq .pg_upmap_items on it, I get valid data back.

So I'm really at a loss as to how to modify the code.

[dvanders]

hi @carlilek could you trying add your line just above upmaps = ... ?

+ osd_dump_json = output.decode('utf-8').strip()
upmaps = json...

[carlilek]

That already appears to be in the code (and it's where the error seems to originate):

# discover existing upmaps
try:
  if use_shell:
    osd_dump_json = subprocess.getoutput('ceph osd dump -f json | jq -r .')
  else:
    cmd = {"prefix": "osd dump", "format": "json"}
    ret, output, errs = cluster.mon_command(json.dumps(cmd), b'', timeout=5)
    osd_dump_json = output.decode('utf-8').strip()
  upmaps = json.loads(osd_dump_json)['pg_upmap_items']
except ValueError:
  eprint('Error loading existing upmaps')
  sys.exit(1)

[JoshuaGabriel]

@carlilek hello, can we get your ceph osd dump --f json to see if we can get a fix for this

[carlilek]

Here you go,
The pythondump.json is the osd_dump_json variable from the script, while the dump.json is the ceph osd dump -f json output.

pythondump.json
dump.json

[JoshuaGabriel]

with Error loading existing upmaps: Expecting value: line 1 column 2345 (char 2344)

root@ceph-test-2:~/testing# cut -c 2288-2348 pythondump.json
"read_balance":{"score_acting":inf,"score_stable":inf,"optima

the inf characters coming from the read balancer seems to be the offender here I think. Could you try with this regex solution to parse it before feeding to json.loads() @carlilek

+ import re 
...
...

+ osd_dump_json = re.sub(r':\s*inf\b', ': "inf"', osd_dump_json) # parse inf to string
upmaps = json.loads(osd_dump_json)['pg_upmap_items']

[carlilek]

Yup, that made it run through. Thank you!

[phineas42]

I experienced this same issue, with a ".mgr" pool reporting "read_balance_score inf", reported as DBL_MAX in the jq output, and unable to be loaded through json.loads.

[dvanders]

I'm testing a solution for this using the yaml library as a more tolerant json parser.

[JoshuaGabriel]

There is also the jq library for parsing, https://pypi.org/project/jq/. IIRC jq on the cli handles the inf keyword correctly.

[afortiorama]

I experience the same the DBL_MAX is unquoted and rejected. It is also in portion of the json that is loaded but not necesary. Perhaps restricting from the start to what is used is a good strategy to minimize problems from json that haven't been sanitized.

[verdurin]

Seeing this too on a Squid cluster.
The regex fix seems to work.

[afortiorama]

Hi,

I have the same problem but with a different string. My dump json contains DBL_MAX in sections that are not used for upmap. IMO if it was possible it would be better to dump only the portions of the jsons that are relevant to the section of the script to reduce the possibility of error.

On the debugging topic I'm not really sure why you hide the real error string behind a not very useful generic statement like 'Error loading existing upmaps' instead of using except ValueError as error: print (error) or print (your_string+error).

thanks