HomeMesosphereNo notifications. 4 unresolved issues.
  • Search
Differential D833

Force delete pod deployment in test.
AbandonedAll Users
Actions

Authored by jeschkies on Jun 6 2017, 4:44 PM.

Details

Reviewers
zen-dog
timcharper
jenkins
Commits
rMARATHON8b1e46a30c60: Retry delete pod deployment in test.
rMARATHON5c150373e198: Force delete pod deployment in test.
JIRA Issues
JIRA MARATHON-7467 MesosAppIntegrationTest Fails With "409 was not equal to 202"
Summary

MesosAppIntegrationTest.MesosApp should deploy a simple pod with health checks
sometimes fails with "409 was not equal to 202" which means the delete
has a conflict. To avoid this error we just force delete.

I did not verify why there is another deployment running. I assume that
there is a slight race condition: We receive the deployment finished
event but the deployment is not completely removed yet.

Test Plan

pipeline

Diff Detail

Repository
rMARATHON marathon
Branch
karsten/409-202-error
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 2863
Build 5433: Marathon (revised)Jenkins
Build 5432: arc lint + arc unit
Changes from before your most recent comment are hidden. Show Older Changes
jeschkies added a comment.Jun 8 2017, 2:29 PM
In D833#32946, @zen-dog wrote:

eventually is also just masking the problem. If the deployment is successful (and it is with waitForDeployment(updateResult)) then you should be able to delete the pod with one call. Everything else is a bug.

It probably. However, it will be super hard to find since this happens roughly 2% of the time. There is no quick fix other than this.

timcharper added a comment.Jun 21 2017, 11:41 PM

I gather the following

  • There is an state consistency issue in which we release the lock for a pod after the deployment affecting said pod is complete
  • We don't have a good way to fix it right now, and it is not an issue likely to be affecting customers right now (deleting a pod right after deployment is probably not a common thing)

Have we already logged a JIRA describing this behavior? It seems like it'd be worth a nice loud comment mentioning "hey there's weird behavior here that we are working around" in the test, and mention this JIRA issue.

Also, @jeschkies - possible to make it only retry in the case of 409 ? It seems like right now it will retry on any failure.

jeschkies abandoned this revision.Jun 28 2017, 3:27 PM

Last 200 hundred runs

 1 "24 was not less than 21"
 1 "24 was not less than 24"
 1 "33 was not less than 31"
 1 "4159 was not less than 4000 the task kill event took longer than the task kill grace period"
 1 "4195 was not less than 4000 the task kill event took longer than the task kill grace period"
 1 "List(ITEnrichedTask(/app-8a693f0c-bcac-429c-ba00-9397df447141,app-8a693f0c-bcac-429c-ba00-9397df447141.3efa83d3-58fb-11e7-9e64-02426214f604,172.16.10.193,Some(List(31514)),Some(2cffc1ac-30fb-4857-961a-e045737b13c9-S0),Some(Sat Jun 24 00:00:00 UTC 2017),Some(Sat Jun 24 00:00:00 UTC 2017),TASK_RUNNING,Some(2017-06-24T16:36:17.385Z))) was not equal to List(ITEnrichedTask(/app-8a693f0c-bcac-429c-ba00-9397df447141,app-8a693f0c-bcac-429c-ba00-9397df447141.406ba004-58fb-11e7-9e64-02426214f604,172.16.10.193,Some(List(31222)),Some(2cffc1ac-30fb-4857-961a-e045737b13c9-S0),Some(Sat Jun 24 00:00:00 UTC 2017),Some(Sat Jun 24 00:00:00 UTC 2017),TASK_RUNNING,Some(2017-06-24T16:36:17.385Z)))"
 1 "List(ITEnrichedTask(/app-c5d6c956-044b-42c3-b014-47649d9acf65,app-c5d6c956-044b-42c3-b014-47649d9acf65.f8b48620-588e-11e7-8371-024285222b1e,172.16.10.183,Some(List(31326)),Some(d72d8f60-8186-4eaa-af24-ed8a5cd82f9d-S0),Some(Sat Jun 24 00:00:00 UTC 2017),Some(Sat Jun 24 00:00:00 UTC 2017),TASK_RUNNING,Some(2017-06-24T03:41:13.814Z))) was not equal to List(ITEnrichedTask(/app-c5d6c956-044b-42c3-b014-47649d9acf65,app-c5d6c956-044b-42c3-b014-47649d9acf65.f9cde561-588e-11e7-8371-024285222b1e,172.16.10.183,Some(List(31963)),Some(d72d8f60-8186-4eaa-af24-ed8a5cd82f9d-S0),Some(Sat Jun 24 00:00:00 UTC 2017),Some(Sat Jun 24 00:00:00 UTC 2017),TASK_RUNNING,Some(2017-06-24T03:41:13.814Z)))"
 1 "List(ITEnrichedTask(/restart-dont-kill,restart-dont-kill.1a0b7415-5847-11e7-8439-02426214f604,172.16.10.193,Some(List(31833)),Some(9f4f66bd-4e57-45cb-aa0c-4048f9a766b6-S0),Some(Fri Jun 23 00:00:00 UTC 2017),Some(Fri Jun 23 00:00:00 UTC 2017),TASK_RUNNING,Some(2017-06-23T19:06:43.071Z))) was not equal to List(ITEnrichedTask(/restart-dont-kill,restart-dont-kill.2fcb149a-5847-11e7-990b-02426214f604,172.16.10.193,Some(List(31295)),Some(9f4f66bd-4e57-45cb-aa0c-4048f9a766b6-S0),Some(Fri Jun 23 00:00:00 UTC 2017),Some(Fri Jun 23 00:00:00 UTC 2017),TASK_RUNNING,Some(2017-06-23T19:06:43.071Z))) Tasks before (List(ITEnrichedTask(/restart-dont-kill,restart-dont-kill.1a0b7415-5847-11e7-8439-02426214f604,172.16.10.193,Some(List(31833)),Some(9f4f66bd-4e57-45cb-aa0c-4048f9a766b6-S0),Some(Fri Jun 23 00:00:00 UTC 2017),Some(Fri Jun 23 00:00:00 UTC 2017),TASK_RUNNING,Some(2017-06-23T19:06:43.071Z)))) and after (List(ITEnrichedTask(/restart-dont-kill,restart-dont-kill.2fcb149a-5847-11e7-990b-02426214f604,172.16.10.193,Some(List(31295)),Some(9f4f66bd-4e57-45cb-aa0c-4048f9a766b6-S0),Some(Fri Jun 23 00:00:00 UTC 2017),Some(Fri Jun 23 00:00:00 UTC 2017),TASK_RUNNING,Some(2017-06-23T19:06:43.071Z)))) abdication are different"
 1 "List(\"app-1.c80b3626-58a7-11e7-9ea9-02426214f604\", \"app-1.d067414e-58a7-11e7-929f-02426214f604\") did not equal List(\"app-1.c80b3626-58a7-11e7-9ea9-02426214f604\", \"app-1.c80fa2f7-58a7-11e7-9ea9-02426214f604\")"
 1 "List(\"app-1.e72b4ade-5996-11e7-9e8c-02426214f604\", \"app-1.efc0dd3c-5996-11e7-b185-02426214f604\") did not equal List(\"app-1.e72a396d-5996-11e7-9e8c-02426214f604\", \"app-1.e72b4ade-5996-11e7-9e8c-02426214f604\")"
 1 "No events matched <Task is declared unreachable>"
 1 "The future returned an exception of type: akka.stream.BindFailedException$, with message: bind failed."
 1 "The test did not complete within the specified 10 seconds time limit."
 1 "\"Pong /[group-8/app-7]\" was not equal to \"Pong /[app-732b33e7-ce58-46ae-a5f9-e68cd5af7f64]\""
 1 "\"Pong /[regression]\" was not equal to \"Pong /[app-ddd39eae-36ee-4868-a0ac-b19f2e2315eb]\""
 1 "\"Pong /[restart-dont-kill]\" was not equal to \"Pong /[app-bb0f2d84-d7cf-4cf3-ac8b-ab2831b73247]\""
 1 "\"Pong /app-[0]\" was not equal to \"Pong /app-[28670201-6b4e-41d5-a0cd-00c0b24bfe1e]\""
 1 "\"Pong /app-[0]\" was not equal to \"Pong /app-[bdf9bd36-bdac-4b44-9c1c-aef06b91d4b4]\""
 1 "\"Pong /app-[1]\" was not equal to \"Pong /app-[b632ba5a-2a3c-4cc2-889d-e38f55ca2960]\""
 1 "deployment visible not valid for 5 seconds. Give up."
 1 "reserved_resources Some({cpus: 0.002, disk: 6.0, gpus: 0.0, mem: 2.0, ports: \"[31735-31735]\" }) did not equal Some({cpus: 0.001, disk: 3.0, gpus: 0.0, mem: 1.0 })"
 1 "used_resources {cpus: 0.002, disk: 6.0, gpus: 0.0, mem: 2.0, ports: \"[31443-31443]\" } did not equal {cpus: 0.001, disk: 3.0, gpus: 0.0, mem: 1.0 }"
 1 "used_resources {cpus: 0.002, disk: 6.0, gpus: 0.0, mem: 2.0, ports: \"[31785-31785]\" } did not equal {cpus: 0.001, disk: 3.0, gpus: 0.0, mem: 1.0 }"
 2 "Tcp command [Connect(localhost:11718,None,List(),Some(10 seconds),true)] failed"
 2 "The test did not complete within the specified 300 seconds time limit."
 3 "The test did not complete within the specified 150 milliseconds time limit."
 4 "No events matched <Replacement task is killed>"
 4 "The future returned an exception of type: mesosphere.marathon.util.TimeoutException, with message: Timed out after 30 seconds."
 9 "Response code was not 201 but 503 with body '{\"message\":\"Futures timed out after [2000 milliseconds]\"}'"
11 "0 was not equal to 1"
21 "No events matched <event deployment_success to arrive>"

The 409 error is gone.

Revision Contents

Diff 3459

View Options

src/test/scala/mesosphere/marathon/integration/MesosAppIntegrationTest.scala

11package mesosphere.marathon
22package integration
33
44import java.util.concurrent.atomic.AtomicInteger
55
6import akka.http.scaladsl.model.HttpResponse
67import mesosphere.marathon.core.health.{ MesosHttpHealthCheck, PortReference }
78import mesosphere.marathon.core.pod.{ HostNetwork, HostVolume, MesosContainer, PodDefinition }
89import mesosphere.marathon.integration.facades.MarathonFacade._
9import mesosphere.marathon.integration.setup.{ EmbeddedMarathonTest, MesosConfig, WaitTestSupport }
10import mesosphere.marathon.integration.setup.{ EmbeddedMarathonTest, MesosConfig, RestResult, WaitTestSupport }
1011import mesosphere.marathon.raml.{ App, Container, DockerContainer, EngineType }
1112import mesosphere.marathon.state.PathId._
1213import mesosphere.{ AkkaIntegrationTest, WhenEnvSet }
14import org.scalatest.concurrent.Eventually
1315
1416import scala.collection.immutable.Seq
1517import scala.concurrent.duration._
1618
1719@IntegrationTest
18class MesosAppIntegrationTest extends AkkaIntegrationTest with EmbeddedMarathonTest {
20class MesosAppIntegrationTest extends AkkaIntegrationTest with EmbeddedMarathonTest with Eventually {
1921
2022 // Integration tests using docker image provisioning with the Mesos containerizer need to be
2123 // run as root in a Linux environment. They have to be explicitly enabled through an env variable.
2224 val envVar = "RUN_MESOS_INTEGRATION_TESTS"
2325
2426 val currentAppId = new AtomicInteger()
2527
2628 // Configure Mesos to provide the Mesos containerizer with Docker image support.
▲ Show 20 Lines • Show All 155 Lines • ▼ Show 20 Line(s)
182184 )
183185 val updateResult = marathon.updatePod(pod.id, updatedPod)
184186
185187 Then("The pod is updated")
186188 updateResult.code should be(200)
187189 waitForDeployment(updateResult)
188190
189191 When("The pod should be deleted")
190 val deleteResult = marathon.deletePod(pod.id)
192 var deleteResult: Option[RestResult[HttpResponse]] = Option.empty
193 eventually {
194 val result = marathon.deletePod(pod.id)
195 result.code should be(202) withClue s"Response: ${result.entityString}" // Delete
196 deleteResult = Some(result)
197 }
191198
192199 Then("The pod is deleted")
193 deleteResult.code should be(202) // Deleted
194 waitForDeployment(deleteResult)
200 waitForDeployment(deleteResult.value)
195201 }
196202
197203 "deploy a pod with Entrypoint/Cmd" taggedAs WhenEnvSet(envVar, default = "true") in {
198204 Given("A pod using the 'hello' image that sets Cmd in its Dockerfile")
199205 val pod = simplePod("simplepod").copy(
200206 containers = Seq(MesosContainer(
201207 name = "hello",
202208 resources = raml.Resources(cpus = 0.1, mem = 32.0),
▲ Show 20 Lines • Show All 174 Lines • Show Last 20 Lines