Skip to content

Add DirectoryListener#764

Open
dr0i wants to merge 1 commit intomasterfrom
702-addDirectoryListener
Open

Add DirectoryListener#764
dr0i wants to merge 1 commit intomasterfrom
702-addDirectoryListener

Conversation

@dr0i
Copy link
Copy Markdown
Member

@dr0i dr0i commented Apr 14, 2026

See #702.

This PR works in principle.
Be aware that there is a bug regarding the WatchService that can result in a loss of awareness of created files, e.g. if these files are created too fast in a row, or e.g. by moving a whole directory with files in it to the directory which is listened to: that directory would be watched, but the files in there won't be recognized. So it's not as inotify in unix contexts - it only comes close to it.

You may want to test it like this:

  1. ./gradlew assembleDist (in the root of this branch to build the runner)
  2. cd ./metafacture-runner/build/distributions/
  3. tar xfz metafacture-core-702-addDirectoryListener-SNAPSHOT-dist.tar.gz
  4. create a FLUX:
echo "default infile = FLUX_DIR + "tmp";
infile|
listen-directory|
print
;"  > directoryListener.flux
  1. mkdir tmp (creates the directory to listen to)
  2. metafacture-core-702-addDirectoryListener-SNAPSHOT-dist/flux.sh directoryListener.flux to execute the FLUX
  3. touch 1 2 3 4 5 6 7 tmp/ to create some files

You should see as output the names of the files with absolute path (which could be given in the FLUX to open-file) and some logs (which are not going to the piped flux-command (as e.g. open-file)) are printed to stdout.
(Interestingly , you see that even "touch"ing 7 files consecutively the WatchService is fast enough to observe the creation of these files.)

If you want the listener to go down: trigger it with the specially named file:
touch shutdownEtlNow

We may want to discuss if it's necessary to improve the behaviour by building some workarounds. One idea would be to just traverse the given directory every n-th second and notate the filenames, if any new appear, to a Map and push these down the pipe. This would guarantee to not miss one file (at the cost of not instantly getting the filename if one was created.)

@dr0i
Copy link
Copy Markdown
Member Author

dr0i commented Apr 14, 2026

functional review: @TobiasNx (and maybe @fsteeg as this PR is supposed to be part of a workflow in the RPB context).
discussion (re bug and workaround): also @blackwinter
code review: @blackwinter (or @fsteeg )

@dr0i dr0i requested review from TobiasNx and fsteeg April 14, 2026 14:41
@dr0i dr0i linked an issue Apr 14, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add possibility to listen on a directory

3 participants