Open
Conversation
Member
Author
|
functional review: @TobiasNx (and maybe @fsteeg as this PR is supposed to be part of a workflow in the RPB context). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See #702.
This PR works in principle.
Be aware that there is a bug regarding the WatchService that can result in a loss of awareness of created files, e.g. if these files are created too fast in a row, or e.g. by moving a whole directory with files in it to the directory which is listened to: that directory would be watched, but the files in there won't be recognized. So it's not as
inotifyin unix contexts - it only comes close to it.You may want to test it like this:
./gradlew assembleDist(in the root of this branch to build the runner)cd ./metafacture-runner/build/distributions/tar xfz metafacture-core-702-addDirectoryListener-SNAPSHOT-dist.tar.gzmkdir tmp(creates the directory to listen to)metafacture-core-702-addDirectoryListener-SNAPSHOT-dist/flux.sh directoryListener.fluxto execute the FLUXtouch 1 2 3 4 5 6 7 tmp/to create some filesYou should see as output the names of the files with absolute path (which could be given in the FLUX to
open-file) and some logs (which are not going to the piped flux-command (as e.g.open-file)) are printed to stdout.(Interestingly , you see that even "touch"ing 7 files consecutively the WatchService is fast enough to observe the creation of these files.)
If you want the listener to go down: trigger it with the specially named file:
touch shutdownEtlNowWe may want to discuss if it's necessary to improve the behaviour by building some workarounds. One idea would be to just traverse the given directory every n-th second and notate the filenames, if any new appear, to a Map and push these down the pipe. This would guarantee to not miss one file (at the cost of not instantly getting the filename if one was created.)