Docs: Add job dispatch and resource tiers documentation

[erick-GeGe]

Description

Please include a summary of the changes, relevant motivation and context.

Issue

• Github Issue ID.

Checklist before requesting a review

• I have performed a self-review of my code.
• My code follows the style guidelines of this project.
• I have made corresponding changes to the documentation.
• New and existing tests pass locally with my changes.
• If this change is a core feature, I have added thorough tests.
• If this change affects or depends on the behavior of other estela repositories, I have created pull requests with the relevant changes in the affected repositories. Please, refer to our official documentation.
• I understand that my pull request may be closed if it becomes obvious or I did not perform all of the steps above.

[joaquingx]

Good coverage of the dispatch system, but I think this reads more like internal code notes than documentation. A few suggestions:

Too much implementation detail

• Things like the Redis lock command (SET spider_jobs_lock 1 NX EX 120), internal function names (_get_cluster_resources(), _dispatch_single_job()), and the "Key Files" table are implementation details that will go stale as code changes. These belong in code comments or docstrings, not in docs.

Suggestion: Simplify to focus on what users need to know (tiers, statuses, config options) and drop the code-level details. The code should document itself.

[joaquingx]

recommended in production is 1000?

[erick-GeGe]

It's not a recommendation — it's the current value in config/settings/prod.py. The default in base.py is 100, but prod.py overrides it to 1000.

[joaquingx]

What about if the DEDICATED_SPIDER_NODES is set to true?

[erick-GeGe]

When DEDICATED_SPIDER_NODES=True, spider pods are scheduled only on nodes labeled with role=<SPIDER_NODE_ROLE>, and _get_cluster_resources() checks capacity only on those nodes. When set to False, pods have no nodeSelector and can land on any node — but the capacity check won't work accurately since it doesn't know which nodes to measure.

[joaquingx]

If this is mandatory why there's an option to deactivate?

[erick-GeGe]

It's not strictly mandatory — it's the recommended setup for production or any infrastructure running many spiders at scale. With DEDICATED_SPIDER_NODES=True, you get accurate capacity checking and isolation between spider workloads and system components. However, for smaller setups where everything runs on one or a few nodes, you may want spiders to be scheduled on any available node, so the option exists for that flexibility.