mavlogdump.py discards parameter messages in tlog files unlike in dataflash

[Maarrk]

When trying to reduce the size of a log like this:

mavlogdump.py --reduce-rate 10 INPUT.tlog --output OUTPUT.tlog --quiet --parms

I lose the parameter messages, as verified by

mavlogdump.py --types PARAM_VALUE INPUT.tlog  # as expected
# versus
mavlogdump.py --types PARAM_VALUE OUTPUT.tlog  # no output

I assume this doesn't happen with DataFlash logs, because of these lines:

pymavlink/tools/mavlogdump.py

Lines 132 to 136 in 2d2d811

	def reduce_rate_msg(m, reduction_rate):
	'''return True if this msg should be discarded by reduction'''
	mtype = m.get_type()
	if mtype in ['PARM','MSG','FMT','FMTU','MULT','MODE','EVT','UNIT', 'VER']:
	return False

---

I have fixed it locally by adding PARAM_VALUE to the list in line 135. I would be happy to contribute a fix, but since we're here I'd like to ensure that feature parity is the desired behaviour and make it complete for all Dataflash types currently in this list:

• PARM: PARAM_VALUE
• MSG: ¿STATUSTEXT?
• FMT: N/A
• FMTU: N/A
• MULT: N/A
• MODE: HEARTBEAT due to custom_mode field
• EVT: unsure, I guess it would be mostly covered by STATUSTEXT
• UNIT: N/A
• VER: ¿AUTOPILOT_VERSION?

N/A: mavlink (thus tlog) has messages with fields and units fully defined by dialect XML files, unlike self-describing dataflash logs.

[shancock884]

My thoughts:
I'd say definitely for PARAM_VALUE, STATUSTEXT, as losing one of these really loses some information (as PARM, MSG would).
AUTOPILOT_VERSION - yes, as equivalent of VER, (but I don't suppose it matters much, as it generally sent as a one-off and its content doesn't change).
For HEARTBEAT I would say not, as this is sent periodically, so if the user wants to reduce the rate, they accept that they may not know mode changes as immediately, in the same way that they reduce rate on other data. (differs from MODE DF msg which is sent once when mode change occurs, so missing it is important).

[Maarrk]

Thank you @shancock884, the criterion of the message being set once looks like a good one to consider here. I agree with you to skip AUTOPILOT_VERSION. Unlike PARAM_VALUE and STATUSTEXT that would likely come in batches and be lost due to rate limiting, AUTOPILOT_VERSION is just a single one, like many other similar messages in common.xml.

However, I think that HEARTBEAT should not be removed not due to flight mode as I originally proposed, but because of its basic role in the protocol. I just had a discussion about how component id cannot be used to determine kind of device anymore and HEARTBEAT is required for most basic tasks. Also this is the only message sent by every device. Let's say you have an autopilot and gimbal tlog, they would have ATTITUDE and GIMBAL_DEVICE_ATTITUDE_STATUS. If you use --reduce-rate 10 you'd keep each of them at no more than 10 Hz. However, since both the autopilot and gimbal send HEARTBEAT at 1 Hz, you might have the autopilot's heart always beat 50ms after gimbal's if their timers are precise and you'd keep all of gimbals messages, and keep none from autopilot.

---

For now I would add: PARAM_VALUE, STATUSTEXT and HEARTBEAT.

What about missions? Just like parameter protocol, it happens rarely so it doesn't increase the log file a lot, but comes in a burst that is obliterated by --reduce-rate. On the other hand CMD message that contains the details of the mission is not included in the current list. Maybe it should be?

[shancock884]

For missions, I think you are right and 'CMD' and 'MISSION_ITEM_INT' should be included (these are the messages used in MAVExplorer.py, cmd_mission function).

Also referring to MAVExplorer.py, the cmd_messages function uses 'MSG','EV','ERR', 'STATUSTEXT'. The list in mavlogdump does not include 'ERR' and appears to have misspelt 'EV' (or maybe it was an older name for the same?)

[Maarrk]

I am almost certain that 'EVT' should be 'EV'. The text EVT is used as abbreviation for "event" in many enumerations, but I can't find any dataflash message with this. The messages are defined in LogStructure.h files for each library, and I can't find it, while EV is defined in AP_Logger itself:

~/repos/ardupilot $ rg 'EV' --glob '**/LogStructure.h'
libraries/AP_Logger/LogStructure.h
724:// @LoggerMessage: EV
1242:    { LOG_EVENT_MSG, sizeof(log_Event), \
1243:      "EV",   "QB",           "TimeUS,Id", "s-", "F-" }, \
1329:    LOG_EVENT_MSG,

libraries/AP_DAL/LogStructure.h
# (not relevant, parts of REV2, REV3 and REVH)
~/repos/ardupilot $ rg 'EVT' --glob '**/LogStructure.h'
# (no output)

EV is also referenced in many more places:

~/repos/ardupilot $ rg "'EV'"
Tools/scripts/playback_log_fg.py
120:        if msg.get_type() == 'EV':

Tools/autotest/arducopter.py
10369:                elif mtype == 'EV':

modules/mavlink/pymavlink/tools/mavlogdump.py
261:            not (m_type in ['FMT', 'FMTU', 'MULT', 'PARM', 'MODE', 'UNIT', 'VER','CMD','MAVC','MSG','EV'] and args.meta)):

modules/mavlink/pymavlink/tools/mavmsgstats.py
26:    'SYSTEM' : ['MAV', 'BAT*', 'EV', 'CMD', 'MODE'],
~/repos/ardupilot $ rg '"EV"'
Tools/autotest/arducopter.py
10360:                m = dfreader.recv_match(type=["GPS", "EV"]) # disarmed

Tools/autotest/vehicle_test_suite.py
10918:            ("Armed EV message",  "EV", lambda e : e.Id == 10),
10919:            ("Disarmed EV message",  "EV", lambda e : e.Id == 11),

libraries/AP_Logger/LogStructure.h
1243:      "EV",   "QB",           "TimeUS,Id", "s-", "F-" }, \

While EVT looks like a mistake here and one other place:

~/repos/ardupilot $ rg "'EVT'"
libraries/AP_Scripting/applets/Aerobatics/FixedWing/dual_log.py
28:types1 = ['ORGN','VEH','PARM','MSG','FMT','FMTU','MULT','MODE','EVT','PTHT']

modules/mavlink/pymavlink/tools/mavlogdump.py
141:    if mtype in ['PARM','MSG','FMT','FMTU','MULT','MODE','EVT','UNIT', 'VER']:
~/repos/ardupilot $ rg '"EVT"'
# (no results)