HomeMesosphereNo notifications. 4 unresolved issues.
  • Search
Differential D963

Change unreachableStrategy to be able to start instant replacement tasks
ClosedAll Users
Actions

Authored by unterstein on Aug 4 2017, 7:34 PM.

Details

Reviewers
jeschkies
zen-dog
jenkins
ichernetsky
kensipe
Commits
rMARATHONd37aa8ed8266: Change unreachableStrategy to be able to start instant replacement tasks
rMARATHON60eccad14a58: Change unreachableStrategy to be able to start instant replacement tasks (#5459)
rMARATHON02b18924ebfb: Change unreachableStrategy to be able to start instant replacement tasks
rMARATHON95273f2e9bf3: fixed test
rMARATHON68f4c38db07d: docs
rMARATHONc14f3cb80e85: [WIP] Change unreachableStrategy to be able to start instant replacement tasks
rMARATHON1dff7962d8f7: adapted validation
rMARATHONd103c6c4938a: Change unreachableStrategy to be able to start instant replacement tasks
rMARATHONc9aeb3e48d5f: docs
rMARATHON47e077ff7761: [WIP] Change unreachableStrategy to be able to start instant replacement tasks
rMARATHON1f979957c9fb: wait for task to be running
JIRA Issues
JIRA MARATHON-7859 Allow immediate replacement of unreachable instances
Test Plan

sbt test
sbt integration:test

Diff Detail

Repository
rMARATHON marathon
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
Changes from before your most recent comment are hidden. Show Older Changes
jeschkies accepted this revision.Aug 9 2017, 2:03 PM

Thanks!

src/test/scala/mesosphere/marathon/integration/TaskUnreachableIntegrationTest.scala
90

Hm, you could add a test with immediate expunge as well ;).

This revision is now accepted and ready to land.Aug 9 2017, 2:03 PM
Closed by commit rMARATHONd103c6c4938a: Change unreachableStrategy to be able to start instant replacement tasks (authored by unterstein). · Explain WhyAug 9 2017, 3:00 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 3978

View Options

changelog.md

Show First 20 Lines • Show All 70 Lines • ▼ Show 20 Line(s)
7171Please move to the `/v2/events` endpoint instead.
7272
7373#### Removed command line parameters
7474- The command line flag `max_tasks_per_offer` has been deprecated since 1.4 and is removed now. Please use `max_instances_per_offer`.
7575
7676#### Deprecated command line parameters
7777- The command line flag `save_tasks_to_launch_timeout` is deprecated and has no effect any longer.
7878
79### New Features
79### Overview
8080
8181#### Networking Improvements Involving Multiple Container Networks
8282
8383The field `networkNames` has been added to [app container's ContainerPortMapping](docs/docs/rest-api/public/api/v2/types/appContainer.raml) and [pod's Endpoint](docs/docs/rest-api/public/api/v2/types/network.raml). Using the field, an app or pod participating in multiple container networks can now forward ports by specifying a single item `networkNames`. For more information, see the [networking documentation](./docs/docs/networking.md).
8484
8585Additionally container port discovery has been improved, with a pod or app being able specify with which container network(s) a port name/protocol/etc is associated. Discovery labels are now generated for container networks associated with ports.
8686
8787#### Mesos Bridge Network Name Configurable
Show All 15 Lines
103103#### Improved Validation Error Messages
104104
105105All validation specified in the RAML is now programatically enforced, leading to more consistent, descriptive, and legible error messages.
106106
107107#### Security improvements
108108
109109Marathon is in better compliance with various security best-practices. An example of this is that Marathon no longer responds to the directory listing request.
110110
111### Fixed issues
112- [MARATHON-7320](https://jira.mesosphere.com/browse/MARATHON-7320) Fix MAX_PER constraint for attributes.
113
114### Overview
115
116111#### File based secrets
117112Marathon has a pluggable interface for secret store providers.
118113Previous versions of Marathon allowed secrets to be passed as environment variables.
119114With this version it is also possible to provide secrets as volumes, mounted under a specified path.
120115See [file based secret documentation](http://mesosphere.github.io/marathon/docs/secrets.html)
121116
117#### Changes around unreachableStrategy
kensipeUnsubmitted
Not Done

IMO this requires more context... we (the developers) have been dealing with this and understand what is happening... however this a change log intended for marathon users. It is highly unlikely they know or understand unreachable (vs lost etc).
at a min add a link to: https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L2053

clarify that this isn't task_lost but TASK_UNREACHABLE. under this condition with default configurations it takes 75 seconds to receive indication from mesos that the task unreachable.

(I did read below and realize you have some details around 75 sec delay... it needs context for the standard reader.

kensipeUnsubmitted
Not Done

I took a stab at a rewrite... (no pride of ownership... use or not what you will)

Changes around unreachableStrategy

Recent changes in Apache Mesos introduced the ability to handle intermittent connectivity to an agent which may be running a Marathon task. This change introduced the TASK_UNREACHABLE. This allows for the ability for a node to disconnect and reconnect to the cluster without having a task replaced. This resulted in (based on default configurations) of a delay of 75 seconds before Marathon would be notified by Mesos to replace the task. The previous behavior of Marathon was usually sub-second replacement of a lost task.

It is now possible to configure unreachableStrategy for apps and pods to instantly* replace unreachable apps or pods. To enable this behavior, you need to configure your app or pod as shown below:

unreachableStrategy: {
    "inactiveAfterSeconds": 0,
    "expunceAfterSeconds": 0
}

Note: Instantly means as soon as marathon becomes aware of the unreachable task. By default marathon is notified after 75 seconds by mesos

that an agent is disconnected. You can change this duration in mesos by configuring `agent_ping_timeout` and `max_agent_ping_timeouts`.
118It is now possible to configure `unreachableStrategy` for apps and pods to instantly* replace unreachable apps or pods. To enable this behavior, you need to configure your app or pod as shown below:
zen-dogUnsubmitted
Not Done

s/not/now

zen-dogUnsubmitted
Not Done

s/instantly/instantly*

119```
120unreachableStrategy: {
121 "inactiveAfterSeconds": 0,
122 "expunceAfterSeconds": 0
123}
124```
125**Note**: Instantly means as soon as marathon becomes aware of the unreachable task. By default marathon is notified after 75 seconds by mesos
zen-dogUnsubmitted
Not Done

may be smth. like:

*: Instantly means as soon as marathon becomes aware...
126 that an agent is disconnected. You can change this duration in mesos by configuring `agent-ping-timeout` and `max_agent_ping_timeouts`.
ichernetskyUnsubmitted
Not Done

In one case dashes are used, in the other one — underscores. Is it correct?

kensipeUnsubmitted
Not Done

replace agent-ping-timeout with agent_ping_timeout
http://mesos.apache.org/documentation/latest/configuration/

127
128### Fixed issues
129- [MARATHON-7320](https://jira.mesosphere.com/browse/MARATHON-7320) Fix MAX_PER constraint for attributes.
122130
123131## Changes from 1.4.1 to 1.4.2
124132Bugfix release
125133
126134### Fixed issues
127135- [MARATHON-4570](https://jira.mesosphere.com/browse/MARATHON-4570) Don't destroy persistent volumes when killing unreachable tasks
128136- [MARATHON-4390](https://jira.mesosphere.com/browse/MARATHON-4390) Lazy event parsing
129137- Improve health checks to log less warnings.
▲ Show 20 Lines • Show All 1931 Lines • Show Last 20 Lines
View Options

docs/docs/configure-task-handling.md

Show All 9 Lines
1010"unreachableStrategy": {
1111 "inactiveAfterSeconds": "<integer>",
1212 "expungeAfterSeconds": "<integer>"
1313}
1414```
1515
1616## `unreachableStrategy` Configuration Options
1717
18- `inactiveAfterSeconds`: If a task instance is unreachable for longer than this value, it is marked inactive and a new instance will launch. At this point, the unreachable task is not yet expunged. The minimum value for this parameter is 1. The default value is 300 seconds.
18- `inactiveAfterSeconds`: If a task instance is unreachable for longer than this value, it is marked inactive and a new instance will launch. At this point, the unreachable task is not yet expunged. The minimum value for this parameter is 0. The default value is 300 seconds.
1919
20- `expungeAfterSeconds`: If an instance is unreachable for longer than this value, it will be expunged. An expunged task will be killed if it ever comes back. Instances are usually marked as unreachable before they are expunged, but that is not required. This parameter must be larger than `unreachableInactiveAfterSeconds`. The default value is 600 seconds.
20- `expungeAfterSeconds`: If an instance is unreachable for longer than this value, it will be expunged. An expunged task will be killed if it ever comes back. Instances are usually marked as unreachable before they are expunged, but that is not required. This parameter must be larger than or equal `unreachableInactiveAfterSeconds`. The default value is 600 seconds.
2121
2222You can use `inactiveAfterSeconds` and `expungeAfterSeconds` in conjunction with one another. For example, if you configure `inactiveAfterSeconds = 60` and `expungeAfterSeconds = 120`, a task instance will be expunged if it has been unreachable for more than 120 seconds and a second instance will be started if it has been unreachable for more than 60 seconds.
2323
2424## Kill Selection
2525You call also define a kill selection to declare whether Marathon kills the youngest or oldest tasks first when rescaling or otherwise killing multiple tasks. The default value for this parameter is `YoungestFirst`. You can also specify `OldestFirst`.
2626
2727Add the `killSelection` parameter to your app definition, or to the `PodSchedulingPolicy` parameter of your pod definition.
2828
Show All 9 Lines
View Options

docs/docs/rest-api/public/api/v2/types/unreachableStrategy.raml

11#%RAML 1.0 Library
22types:
33 UnreachableStrategy: (UnreachableDisabled | UnreachableEnabled)
44 UnreachableDisabled:
55 type: string
66 enum: [ disabled ]
77 default: disabled
88 UnreachableEnabled:
99 type: object
1010 properties:
1111 inactiveAfterSeconds?:
1212 type: integer
1313 format: int64
1414 default: 300
15 minimum: 1
15 minimum: 0
1616 description: |
1717 If an instance is unreachable for longer than inactiveAfter seconds it is marked
1818 as inactive. This will trigger a new instance launch. The original task is not
1919 expunged yet. Must be less than expungeAfterSeconds.
2020
2121 The default value is set to 5 minutes (300 seconds).
2222
2323 expungeAfterSeconds?:
2424 type: integer
2525 format: int64
2626 default: 600
27 minimum: 1
27 minimum: 0
2828 description: |
2929 If an instance is unreachable for longer than unreachableExpungeAfter seconds it will be expunged. That means
3030 it will be killed if it ever comes back. Instances are usually marked as unreachable before they are expunged
3131 but they don't have to. This value is required to be greater than inactiveAfterSeconds.
3232
3333 The default value is set to 10 minutes (600 seconds).
View Options

src/main/scala/mesosphere/marathon/state/UnreachableStrategy.scala

Show All 29 Lines
3030 build
3131}
3232object UnreachableEnabled {
3333 val DefaultInactiveAfter: FiniteDuration = 5.minutes
3434 val DefaultExpungeAfter: FiniteDuration = 10.minutes
3535 val default = UnreachableEnabled()
3636
3737 implicit val unreachableEnabledValidator = validator[UnreachableEnabled] { strategy =>
38 strategy.inactiveAfter should be >= 1.second
39 strategy.inactiveAfter should be < strategy.expungeAfter
38 strategy.inactiveAfter should be >= 0.second
39 strategy.inactiveAfter should be <= strategy.expungeAfter
4040 }
4141}
4242
4343object UnreachableStrategy {
4444
4545 def default(resident: Boolean = false): UnreachableStrategy = {
4646 if (resident) UnreachableDisabled else UnreachableEnabled.default
4747 }
Show All 19 Lines
View Options

src/test/scala/mesosphere/marathon/api/validation/AppDefinitionValidationTest.scala

11package mesosphere.marathon
22package api.validation
33
44import com.wix.accord.scalatest.ResultMatchers
55import mesosphere.UnitTest
66import mesosphere.marathon.core.plugin.PluginManager
77import mesosphere.marathon.state._
88import org.apache.mesos.{ Protos => Mesos }
99
10import scala.concurrent.duration._
11
1210class AppDefinitionValidationTest extends UnitTest with ResultMatchers {
1311
1412 "AppDefinition" when {
1513 "created with the default unreachable strategy" should {
1614 "be valid" in new Fixture {
1715 val app = AppDefinition(id = PathId("/test"), cmd = Some("sleep 1000"))
1816
1917 val validation = validator(app)
2018 validation shouldBe aSuccess
2119
22 }
23 }
24
25 "created with an invalid unreachable strategy" should {
26 "be invalid" in new Fixture {
27 val app = AppDefinition(
28 id = PathId("/test"),
29 cmd = Some("sleep 1000"),
30 unreachableStrategy = UnreachableEnabled(0.seconds))
31
32 val expectedViolation = GroupViolationMatcher(description = "unreachableStrategy", constraint = "is invalid")
33 validator(app) should failWith(expectedViolation)
3420 }
3521 }
3622
3723 "created with dependencies" should {
3824
3925 "be valid with dependencies pointing to a a subtree of this app" in new Fixture {
4026 val app = AppDefinition(
4127 id = PathId("/a/b/c/d"),
▲ Show 20 Lines • Show All 90 Lines • Show Last 20 Lines
View Options

src/test/scala/mesosphere/marathon/integration/TaskUnreachableIntegrationTest.scala

11package mesosphere.marathon
22package integration
33
44import mesosphere.AkkaIntegrationTest
55import mesosphere.marathon.integration.facades.ITEnrichedTask
66import mesosphere.marathon.integration.setup._
7import mesosphere.marathon.raml.App
78import mesosphere.marathon.state.PathId._
89import org.scalatest.Inside
910
11import scala.concurrent.duration._
12
1013@IntegrationTest
1114class TaskUnreachableIntegrationTest extends AkkaIntegrationTest with EmbeddedMarathonTest with Inside {
1215
1316 override lazy val mesosNumMasters = 1
1417 override lazy val mesosNumSlaves = 2
1518
1619 override val marathonArgs: Map[String, String] = Map(
1720 "reconciliation_initial_delay" -> "5000",
▲ Show 20 Lines • Show All 59 Lines • ▼ Show 20 Line(s)
7780 matchEvent("TASK_KILLED", replacement)
7881 }
7982
8083 And("there is only one running task left")
8184 marathon.tasks(app.id.toPath).value should have size 1
8285 marathon.tasks(app.id.toPath).value.head.state should be("TASK_RUNNING")
8386 }
8487
88 "A task unreachable update with inactiveAfterSeconds 0 will trigger a replacement task instantly" in {
89 Given("a new app with proper timeouts")
90 val strategy = raml.UnreachableEnabled(inactiveAfterSeconds = 0, expungeAfterSeconds = 60)
jeschkiesUnsubmitted
Not Done

Hm, you could add a test with immediate expunge as well ;).

91 val app = appProxy(testBasePath / "unreachable-instant", "v1", instances = 1, healthCheck = None).copy(
92 unreachableStrategy = Option(strategy)
93 )
94 waitForDeployment(marathon.createAppV2(app))
95 val task = waitForTasks(app.id.toPath, 1).head
96
97 When("the slave is partitioned")
98 mesosCluster.agents(0).stop()
99 mesosCluster.agents(1).start() // Start an alternative agent
100
101 Then("the task is declared unreachable")
102 waitForEventMatching("Task is declared unreachable") {
103 matchEvent("TASK_UNREACHABLE", task)
104 }
105
106 Then("the replacement task is running")
107 // wait not longer than 1 second, because it should be replaced even faster
108 waitForEventMatching("Replacement task is declared running", 1.seconds) {
jeschkiesUnsubmitted
Done

Hm, this looks like the issue. The unreachable task is expunged rather fast. How about waiting for the kill and a new running task?

untersteinAuthorUnsubmitted
Not Done

woop woop

109 matchEvent("TASK_RUNNING", app)
110 }
111
112 // immediate replacement should be started
113 val tasks = marathon.tasks(app.id.toPath).value
114 tasks should have size 2
115 tasks.groupBy(_.state).keySet should be(Set("TASK_RUNNING", "TASK_UNREACHABLE"))
jeschkiesUnsubmitted
Not Done

This seems racy as well. The unreachable task might be expunged, right?

116 }
117
85118 // regression test for https://github.com/mesosphere/marathon/issues/4059
86119 "Scaling down an app with constraints and unreachable task will succeed" in {
87120 Given("an app that is constrained to a unique hostname")
88121 val constraint = Set(Seq("node", "MAX_PER", "1"))
89122
90123 // start both slaves
91124 mesosCluster.agents.foreach(_.start())
92125
Show All 38 Lines
131164 eventually(inside(marathon.tasks(app.id.toPath).value) {
132165 case t :: Nil =>
133166 t.state shouldBe "TASK_RUNNING"
134167 })
135168
136169 marathon.listDeploymentsForBaseGroup().value should have size 0
137170 }
138171 }
172
173 def matchEvent(status: String, app: App): CallbackEvent => Boolean = { event =>
174 event.info.get("taskStatus").contains(status) &&
175 event.info.get("appId").contains(app.id)
176 }
177
139178 def matchEvent(status: String, task: ITEnrichedTask): CallbackEvent => Boolean = { event =>
140179 event.info.get("taskStatus").contains(status) &&
141180 event.info.get("taskId").contains(task.id)
142181 }
143182
144183 private def matchDeploymentStart(appId: String): CallbackEvent => Boolean = { event =>
145184 val infoString = event.info.toString()
146185 event.eventType == "deployment_info" && matchScaleApplication(infoString, appId)
147186 }
148187
149188 private def matchScaleApplication(infoString: String, appId: String): Boolean = {
150189 infoString.contains(s"List(Map(actions -> List(Map(action -> ScaleApplication, app -> $appId)))))")
151190 }
152191}
View Options

src/test/scala/mesosphere/marathon/state/UnreachableStrategyTest.scala

Show All 23 Lines
2424 "sees disabled as valid" in {
2525 val strategy = UnreachableDisabled
2626 validate(strategy) shouldBe aSuccess
2727 }
2828
2929 "fail when time until expunge is smaller" in {
3030 val strategy = UnreachableEnabled(inactiveAfter = 2.seconds, expungeAfter = 1.second)
3131 validate(strategy) should failWith(
32 "inactiveAfter" -> "got 2 seconds, expected less than 1 second")
32 "inactiveAfter" -> "got 2 seconds, expected 1 second or less")
3333 }
3434
3535 "fail when time until expunge is equal to time until inactive" in {
3636 val strategy = UnreachableEnabled(inactiveAfter = 2.seconds, expungeAfter = 2.seconds)
37 validate(strategy) should failWith(
38 "inactiveAfter" -> "got 2 seconds, expected less than 2 seconds")
37 validate(strategy) shouldBe aSuccess
3938 }
4039 }
4140
4241 "toProto" should {
4342 Seq(
4443 UnreachableDisabled,
4544 UnreachableEnabled(inactiveAfter = 10.seconds, expungeAfter = 20.seconds)).foreach { unreachableStrategy =>
4645
4746 s"round trip serializes ${unreachableStrategy}" in {
4847 UnreachableStrategy.fromProto(unreachableStrategy.toProto) shouldBe (unreachableStrategy)
4948 }
5049 }
5150 }
5251}