Export keypresses with synchronized timestamps for external tools

[pkazmier]

Is there any appetite for a PR that adds a command line flag --keylog flag to record keypresses with timestamps as JSON to a file for use by external subtitle generation or other accessibility tools. I was going to submit a PR, but the checklist in the PR template asked that a discussion be started first. The code has been written.

I love 'vhs' and the idea of reproducible screencasts, but I need the ability to overlay keypresses for viewers. While there is PR #496, it has been inactive for 2 years and adds quite a bit of complexity to the codebase. So I thought about taking an alternate, more minimal, approach to simply allow the exporting of a keypress log that other tools could use for captioning. With this change, I could build a separate tool to add captions using 'ffmpeg'.

As for timings, there is another PR #616 that started this effort. However, in writing my original version, I attempted to use wall clock as well, but this is not sufficient for captioning of keypresses due to the amount of drift. In my testing, by the time I got to the end of a 30s clip, the keypress events were off by 1.35s. So I took a different approach and generated timestamps based on the current frame being captured. This improved accuracy significantly.

Here is the output from three runs of the same tape file that results in a 45s video. Below are the last keypress events from each run to illustrate the accuracy of the timestamps. In each of the three cases, I reviewed the output video frame by frame to measure the drift: 80ms, 80ms, and 60ms. Far better than 1,350ms when using wall clock.

The output of JSON is simple by design. It contains the key and milliseconds relative to start of recording. This makes it easy for 3rd party tools to process the files without having to parse a time format that no one will agree upon 😄 .

I even added log message showing creation of the keylog:

I would like the opportunity to submit the PR for review as it would allow others to build tools to add captioning. The changes are minimal to 'vhs' codebase with only 278 lines of additions—no changes or deletes. The bulk of new code is confined to 'keylogger.go' (86 lines) and its companion test file 'keylogger_test.go' (168 lines). There are only 20 additions (excluding blank lines and comments) to the rest of 'vhs'. In addition, there are no changes to existing API or behavior.

[pkazmier]

And today I created a small script to consume the key log to generate captions. Here is a demo.

demo-captioned.mp4

[pkazmier]

Here is the CLI tool I wrote to add the captions using the exported key log from my patched version of VHS.

[pkazmier]

I've decided to add captioning directly to VHS. Please see PR #719 for lots of demos.