6 scheduling

[vlerkin]

What happens in the PR:

1. The logic of event watcher was separated in an observer class; the logic of log watching stayed in a log handler class, but the initialization was changed to subscribe to the event in case jobless feature was configured;
2. The new class KubernetesScheduler was created to handle logic when jobs must be unsuspended and how (ordered);
3. scheduler endpoint was modified, logic to set a value for start_suspended parameter was added;
4. schedule method from k8s launcher has a new start_suspended parameter, it's value is passed when called inside the api; also new methods were added: unsuspend_job patches existing suspended job suspend=False, get_running_jobs_count returns the number of jobs that are currently running, list_suspended_jobs returns the list of jobs where spec.suspend is true, _get_job_name extracts the job name from the metadata, it is then used for unsuspend function;

The big picture:
Event watcher connects to the k8s api and receives the stream of events, it then notifies the subscribers if a new event is received and passes it to the provided callback. The subscriber - KubernetesScheduler - receives event in a handle_pod_event method, this method reacts to the changes in job statuses, and if job completed running or failed it calls another method - check_and_unsuspend_jobs - that checks capacity and unsuspends jobs until the number of allowed parallel jobs is reached, while doing this it relies on another method - get_next_suspended_job_id - to unsuspend the most recent job, to keep the order in which jobs were initially scheduled.
When the job is scheduled, based on the number of currently active jobs and max_proc provided in the config (default is 4), the job runs or goes to the queue of suspended jobs (native k8s queue). Then events that change the number of active jobs trigger the logic of KubernetesScheduler class that unsuspend suspended jobs until the desired state (num of parallel jobs) is achieved.

[wvengen]

Ah, nice you were able to come up with something so quickly already!
I looked at it from a high level, and noticed that this is currently implemented for Kubernetes only (that makes sense), and also setup in such a way that it needs refactoring for Docker. I would think of the scheduler as something that could work for both Docker and Kubernetes, especially the scheduling decisions. Also, there is now k8s-specific code in the main file (e.g. the import), and the kubernetes scheduler, this makes the code somewhat spaghetti: there are specific implementation-specific classes where responsibility is meant to be delegated. If you need to access the scheduler in the main file, use a generic scheduler, and make the docker-based parts not implemented. I think that would give a much cleaner design.

Also, I would consider making the launcher responsible for scheduling. And then have the scheduler talk to the launcher to actually start jobs.

I'm not yet sure if we should allow running without the scheduler, or if it would always be active.

[wvengen]

Hope my feedback was at an angle that helps you at this stage. In any case, well done, keep it going!

p.s. the CI error looks like it could be cause by Kubernetes-specific things having entered into the main api code, which wouldn't work when running with Docker.

[vlerkin]

Working on Docker implementation to be added to this PR

[wvengen]

Great to see a working version! Quite readable :)
I think it needs a little cleanup, but you're getting there, I think.

[vlerkin]

I have problems because I separated this PR partially and now I have a multiverse which I need to refactor to the only source of truth. Going to spend some uncertain amount of time on that.

[wvengen]

The way I would do this:

1. Continue working on this PR, until you need the functionality developed in the other PR (or until it has been merged).
2. Interactive rebase on the branch of the other PR. Filter out the commits you had here that you rewrote in the other branch.
3. There may be little or much work to do in resolving conflicts. If it is really many, in various commits, you may consider another route (see below).
4. Test, done.

Of this is much work in many commits, you may consider first doing an interactive rebase of this PR, to simplify it, and reduce the number of commits (that each may need amending).

Yes, this is a bit of work, but something I come across now and then, in various projects.
Sorry for the complexity!

[vlerkin]

Thank you for the advice!
I was thinking of dropping the commit with the merging main to this branch, then make the code work so the tests run if needed. Then merging with that other branch that refactored the observer further and make the code of both branches work together and then check if there are any conflicts with main and resolving those. This is a bit longer way than simply redoing the merge with the main branch but I messed up the last one because I lost track of changes, so gradually rebuilding this branch is a bit easier for me.

No worries, this is me who messed up merging, complexity is part of the job:D Learning to make more granular commits and cleaner PRs the hard way:D

[vlerkin]

I modified one of the methods in the scheduler (get_next_suspended_job_id) to handle cases if a job does not have a creation_timestamp. It is not expected but if someone used a custom resource and forgot to add this field or made any other error, the job will get the timestamp assigned and will be processed like other jobs in the queue.

Also, there are now unit tests that cover different scenarios for the scheduler.

If you have any other comments for improvements, let me know!

[wvengen]

Could you please resolve conflicts on main, so that I can see what this PR specifically changes?

[wvengen]

At first glance, well done! Seems to do the job (though haven't tested it).
An integration test would increase my confidence that it performs well (now I'd have to run it locally to see if it actually works - I think it would, but still I would feel necessitated to do so).

Also, I think the scheduling logic is now implemented in K8s and Docker separately. Would it make sense to have a single piece of code decide when to schedule, and let e.g. the launcher and listener be the interface to K8s/Docker? Haven't thought this fully through, but the question comes up.

Some questions and notes remain, otherwise it's well on the way.

[vlerkin]

The launcher is already sort of an interface, since every implementation uses its own launcher. Listener cannot be an interface because event watcher is a native Kubernetes thing, if I understand your comment correctly. So for k8s we still use the watcher to start suspended jobs when at least one job is done or deleted, and for Docker we have a background thread that checks the state of existing containers, we have discussed it previously.

[wvengen]

Rebased on main, adapted integration tests to setup with different configuration files.

[wvengen]

I'm not fully happy with the current integration tests, using a shellscript to patch the k8s setup. It might be cleaner to add a YAML file with the desired k8s manifest (in this case, for the role), and let the CI script save the cluster state on cluster setup, and restore it before running a test (kubectl apply -f first the pristine cluster state, then the test-specific resource; would even save a scale down as it is part of the pristine cluster state - but not the waiting on it, so perhaps remains useful to keep scale down).
Downside would be that the full role state is still necessary, so any change to scrapyd-k8d's required roles need to be included in the test-specific role manifest as well, so in that respect, a patch is actually more to the point.
So perhaps it is fine as it is, just wanted to share my thoughts here.

[vlerkin]

Wait until the second is finished too

listjobs_wait(jobid2, 'finished', max_wait=STATIC_SLEEP+MAX_WAIT)

Just curious why would you wait until the second job is done? If it was scheduled after the first job is done then the feature works properly, I am not sure what we are testing here with the waiting for the second one.

[vlerkin]

I would group the unit tests under 4 classes:
TestKubernetesSchedulerInitialization: test_k8s_scheduler_init, test_k8s_scheduler_init_invalid_max_proc

TestPodEventHandling:
test_handle_pod_event_with_non_dict_event,
test_handle_pod_event_pod_missing_status, etc

TestJobSuspensionManagement: test_check_and_unsuspend_jobs_with_capacity_and_suspended_jobs, test_check_and_unsuspend_jobs_no_suspended_jobs, etc

TestSuspendedJobSelection: test_get_next_suspended_job_id_with_suspended_jobs, test_get_next_suspended_job_id_no_suspended_jobs, etc

This is much more readable and easier to contribute to add or change tests. What do you think?

[vlerkin]

And to make code more compact, we could use @pytest.mark.parametrize to avoid code duplications

[wvengen]

Hi 👋 Thanks for your input!

Just curious why would you wait until the second job is done? If it was scheduled after the first job is done then the feature works properly, I am not sure what we are testing here with the waiting for the second one.

As a general note, the integration tests are, integration tests, and meant to test the system as a whole. Here it means that it is not strictly necessary to wait on the second job, but it is still part of the expected flow, and doesn't hurt to test. These tests are more like what a user would expect when using the system; not handling specific edge cases very isolated, but may include edge cases in the integrated flow.

TestKubernetesSchedulerInitialization, TestPodEventHandling, TestJobSuspensionManagement, TestSuspendedJobSelection

These tests sound Kubernetes-specific, and are not really about testing a full interaction cycle with scrapyd-k8s using its API only. Therefore they don't really belong in the (current) integration tests, I think.

There are probably specific cases to cover, as you write. Very useful to know about. Maybe we could express them as a full integration tests, that triggers a certain corner-case, and should work in a certain way, regardless of backend (k8s/docker).

If we want to test the launchers (incl. the possible schedulers), then we'd need another kind of tests, perhaps docker- and k8s-specific tests, that check how a cluster/node responds to launcher commands, and vice versa. Here we might test the surface API of the launcher (instead of the REST API). This is a kind of test we don't have yet (all tests are now backend-agnostic).

In the early stages of this project, to keep testing work managable, no backend-specific tests were created. There may come a time where this project grows, and needs backend-specific tests, but I see the overhead as a bit too much as of now - as long as we can cover enough ground with the backend-independent integration tests.

test_k8s_scheduler_init_invalid_max_proc

This seems like it would be a separate integration test with a config file having an invalid value. I think the API does not expose this, so either the daemon does not start at all, or runs with a reduced feature set. Both are not reported through the API, so cannot be well tested now. So this requires a different kind of testing setup, curently out-of-scope. I think this would belong in a different issue, in revising the testing infrastructure.

Note that I'd really like to keep a distinction between running actual tests, and setting up the daemon under test in an environment.

[vlerkin]

These tests sound Kubernetes-specific, and are not really about testing a full interaction cycle with scrapyd-k8s using its API only. Therefore they don't really belong in the (current) integration tests, I think.

They are unit tests, I just looked at them again and suggested possible improvements, it was probably a bit confusing:)

[vlerkin]

I don't really see any improvements for the integration tests, looks good as it is

[vlerkin]

Hey Willem, is there anything you would like to add to this PR?

[wvengen]

Some small notes, but I think it is mostly ready to merge! 🎉

[vlerkin]

I made changes you mentioned, could you, please, check if this is what you expected?

[wvengen]

Almost there!

It's always good to take a fresh look at the PR yourself, after you haven't looked it at for a while. And then you're still bound to miss some things that have become too familiar. Thanks for taking the time to finish the last bits!

[wvengen]

Thank you!