should call reviveOffer in fine-grained mesos mode after tasks successfully finished #816
Open
xiajunluan wants to merge 1 commit intomesos:masterfrom
Open
should call reviveOffer in fine-grained mesos mode after tasks successfully finished #816xiajunluan wants to merge 1 commit intomesos:masterfrom
xiajunluan wants to merge 1 commit intomesos:masterfrom
Conversation
|
Thank you for your pull request. An admin will review this request soon. |
Member
|
In this mode, Mesos itself should send back offers when our refusals time out after 1 second. Also, we call reviveOffers when a TaskSetManager is submitted. What kind of performance difference have you seen with this, and with what Mesos version? Calling reviveOffers on every task finish can also be quite expensive because Mesos will list all the free nodes in the cluster for us. |
Member
|
OK, I'd like to ask the Mesos guys about this, because I think calling reviveOffers on every task-finished will be very expensive on a larger cluster. In older versions of Mesos, the system made a new resource offer as soon as a task finished. Has this changed recently? CC @benh |
xiajunluan
pushed a commit
to xiajunluan/spark
that referenced
this pull request
May 30, 2014
Some improvements to MLlib guide: 1. [SPARK-1872] Update API links for unidoc. 2. [SPARK-1783] Added `page.displayTitle` to the global layout. If it is defined, use it instead of `page.title` for title display. 3. Add more Java/Python examples. Author: Xiangrui Meng <meng@databricks.com> Closes mesos#816 from mengxr/mllib-doc and squashes the following commits: ec2e407 [Xiangrui Meng] format scala example for ALS cd9f40b [Xiangrui Meng] add a paragraph to summarize distributed matrix types 4617f04 [Xiangrui Meng] add python example to loadLibSVMFile and fix Java example d6509c2 [Xiangrui Meng] [SPARK-1783] update mllib titles 561fdc0 [Xiangrui Meng] add a displayTitle option to global layout 195d06f [Xiangrui Meng] add Java example for summary stats and minor fix 9f1ff89 [Xiangrui Meng] update java api links in mllib-basics 7dad18e [Xiangrui Meng] update java api links in NB 3a0f4a6 [Xiangrui Meng] api/pyspark -> api/python 35bdeb9 [Xiangrui Meng] api/mllib -> api/scala e4afaa8 [Xiangrui Meng] explicity state what might change
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[Spark-872]I think we should call reviveOffer in statusUpdate function to request resource, In scheduler.statusUpdate function, it calls reviveOffer only for TASK_LOST and TASK_FAILED, so it need deal with TASK_FINISHED scenario, this improvement will enhance performance significantly.