Fix flaky MesosAppIntegrationTest
ClosedAll Users
Actions

"url": "https://downloads.mesosphere.io/marathon/snapshots/marathon-1.5.0-SNAPSHOT-667-g6c05db4.tgz",
"sha1": "903cf897283c2927e199e342a440a121e208edda"

＼\ ٩( ᐛ )و /／

Harbormaster completed building B3160: Diff 3841.Jul 25 2017, 4:57 PM

Closed by commit rMARATHON85c41bc279b6: Fix flaky MesosAppIntegrationTest (authored by zen-dog). · Explain WhyJul 25 2017, 5:01 PM

This revision was automatically updated to reflect the committed changes.

src/test/scala/mesosphere/marathon/integration/setup/MarathonTest.scala
534	this func doesn't actually need to accept a mutable Map. The parameter could easily be an immutable map. If a mutable structure is really needed, it then this func should convert the Map param from immutable to mutable. Exposing an API that accepts mutable things is something that we should strongly discourage except in very, very rare circumstances. IMO there is not justification here for doing so.
535	nit: public func, should have a return type

src/test/scala/mesosphere/marathon/integration/setup/MarathonTest.scala
534	JIRA MARATHON-7655 Address review feedback for D935 integration test changes

src/test/scala/mesosphere/marathon/integration/setup/MarathonTest.scala
534	I thought about taking in an `immutable.Map` and converting it to `mutable` but it's a internal tester helper and I decided it's not worth extra effort.

Revision Contents

			Path	Packages
M			src/main/scala/mesosphere/marathon/core/group/impl/GroupManagerImpl.scala (2 lines)
M			src/test/scala/mesosphere/marathon/integration/AppDeployIntegrationTest.scala (11 lines)
M			src/test/scala/mesosphere/marathon/integration/MesosAppIntegrationTest.scala (24 lines)
M			src/test/scala/mesosphere/marathon/integration/setup/MarathonTest.scala (18 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	3834	071e6e2		Jul 24 2017, 4:26 PM	★	★
Diff 2	3841	6c05db4	- Review feedback	Jul 25 2017, 12:02 PM	★	★
Diff 3	3845	6c05db4	rMARATHON85c41bc279b6894073a1ecb11d7466d3403ac520	Jul 25 2017, 5:00 PM	★	★

Commit	Tree	Parents	Author	Summary	Date
f60522c91744	a4ba6f012afd	f5c15bf0ff18	Aleksey Dukhovniy	Review feedback	Jul 25 2017, 12:01 PM
f5c15bf0ff18	5b17e4939420	6c05db4490e0	Aleksey Dukhovniy	Fix flaky MesosAppIntegrationTest (Show More…)	Jul 24 2017, 4:23 PM

Diff 3845

View Options

src/main/scala/mesosphere/marathon/core/group/impl/GroupManagerImpl.scala

Show First 20 Lines • Show All 105 Lines • ▼ Show 20 Line(s)
106	106	logger.info(s"Updated groups/apps/pods according to plan ${plan.id}")
107	107	// finally update the root under the write lock.
108	108	root := plan.target
109	109	plan
110	110	}
111	111	}
112	112	deployment.onComplete {
113	113	case Success(plan) =>
114		logger.info(s"Deployment acknowledged. Waiting to get processed:\n$plan")
	114	logger.info(s"Deployment ${plan.id}:${plan.version} for ${plan.target.id} acknowledged. Waiting to get processed")
115	115	eventStream.publish(GroupChangeSuccess(id, version.toString))
116	116	case Failure(ex: AccessDeniedException) =>
117	117	// If the request was not authorized, we should not publish an event
118	118	logger.warn(s"Deployment failed for change: $version", ex)
119	119	case Failure(NonFatal(ex)) =>
120	120	logger.warn(s"Deployment failed for change: $version", ex)
121	121	eventStream.publish(GroupChangeFailed(id, version.toString, ex.getMessage))
122	122	}
▲ Show 20 Lines • Show All 83 Lines • Show Last 20 Lines

View Options

src/test/scala/mesosphere/marathon/integration/AppDeployIntegrationTest.scala

Show First 20 Lines • Show All 668 Lines • ▼ Show 20 Line(s)
669	669
670	670	When("the deployment is rolled back")
671	671	val delete = marathon.deleteDeployment(deploymentId, force = false)
672	672	delete should be(OK)
673	673	val rollbackId = delete.originalResponse.headers.find(_.name == RestResource.DeploymentHeader).getOrElse(throw new IllegalArgumentException("No deployment id found in Http Header")).value()
674	674
675	675	Then("old deployment should be canceled and rollback-deployment succeed")
676	676	// Both deployment events may come out of order
677		val waitingFor = mutable.Map("deployment_failed" -> deploymentId, "deployment_success" -> rollbackId)
678		waitForEventMatching(s"waiting for canceled $deploymentId and successful $rollbackId") { event =>
679		if (waitingFor.get(event.eventType).fold(false)(_ == event.id)) {
680		waitingFor -= event.eventType
681		}
682		waitingFor.isEmpty
683		}
	677	val waitingFor = mutable.Map[String, CallbackEvent => Boolean](
	678	"deployment_failed" -> (_.id == deploymentId),
	679	"deployment_success" -> (_.id == rollbackId))
	680	waitForEventsWith(s"waiting for canceled $deploymentId and successful $rollbackId", waitingFor)
684	681
685	682	Then("no more deployment in the queue")
686	683	WaitTestSupport.waitUntil("Deployments get removed from the queue") {
687	684	marathon.listDeploymentsForBaseGroup().value.isEmpty
688	685	}
689	686
690	687	Then("the app should also be gone")
691	688	marathon.app(id) should be(NotFound)
▲ Show 20 Lines • Show All 139 Lines • Show Last 20 Lines

View Options

src/test/scala/mesosphere/marathon/integration/MesosAppIntegrationTest.scala

1	1	package mesosphere.marathon
2	2	package integration
3	3
4	4	import java.util.concurrent.atomic.AtomicInteger
5	5
6		import mesosphere.marathon.core.health.{ MesosHttpHealthCheck, PortReference }
	6	import mesosphere.marathon.api.RestResource
	7	import mesosphere.marathon.core.health.{MesosHttpHealthCheck, PortReference}
7	8	import mesosphere.marathon.core.pod._
8	9	import mesosphere.marathon.integration.facades.MarathonFacade._
9		import mesosphere.marathon.integration.setup.{ EmbeddedMarathonTest, MesosConfig, WaitTestSupport }
10		import mesosphere.marathon.raml.{ App, Container, DockerContainer, EngineType }
	10	import mesosphere.marathon.integration.setup.{EmbeddedMarathonTest, MesosConfig, WaitTestSupport}
	11	import mesosphere.marathon.raml.{App, Container, DockerContainer, EngineType}
11	12	import mesosphere.marathon.state.PathId._
12		import mesosphere.{ AkkaIntegrationTest, WhenEnvSet }
	13	import mesosphere.{AkkaIntegrationTest, WhenEnvSet}
13	14
14	15	import scala.collection.immutable.Seq
	16	import scala.collection.mutable
15	17	import scala.concurrent.duration._
16	18
17	19	@IntegrationTest
18	20	class MesosAppIntegrationTest extends AkkaIntegrationTest with EmbeddedMarathonTest {
19	21
20	22	// Integration tests using docker image provisioning with the Mesos containerizer need to be
21	23	// run as root in a Linux environment. They have to be explicitly enabled through an env variable.
22	24	val envVar = "RUN_MESOS_INTEGRATION_TESTS"
▲ Show 20 Lines • Show All 266 Lines • ▼ Show 20 Line(s)
289	291	Given("a pod with constraints that cannot be fulfilled")
290	292	val constraint = Protos.Constraint.newBuilder().setField("nonExistent").setOperator(Protos.Constraint.Operator.CLUSTER).setValue("na").build()
291	293	val pod = simplePod("simplepod").copy(
292	294	constraints = Set(constraint)
293	295	)
294	296
295	297	val createResult = marathon.createPodV2(pod)
296	298	createResult should be(Created)
297		val deploymentId = createResult.originalResponse.headers.find(_.name == "Marathon-Deployment-Id").map(_.value)
	299	val deploymentId = createResult.originalResponse.headers.find(_.name == RestResource.DeploymentHeader).map(_.value)
298	300	deploymentId shouldBe defined
299	301
300	302	Then("the deployment gets created")
301	303	WaitTestSupport.validFor("deployment visible", 5.second)(marathon.listDeploymentsForBaseGroup().value.size == 1)
302	304
303	305	When("the deployment is deleted")
304	306	val deleteResult = marathon.deleteDeployment(deploymentId.get, force = true)
305	307	deleteResult should be(Deleted)
Show All 12 Lines
318	320	Given("a pod with constraints that cannot be fulfilled")
319	321	val constraint = Protos.Constraint.newBuilder().setField("nonExistent").setOperator(Protos.Constraint.Operator.CLUSTER).setValue("na").build()
320	322	val pod = simplePod("simplepod").copy(
321	323	constraints = Set(constraint)
322	324	)
323	325
324	326	val createResult = marathon.createPodV2(pod)
325	327	createResult should be(Created)
326		val deploymentId = createResult.originalResponse.headers.find(_.name == "Marathon-Deployment-Id").map(_.value)
	328	val deploymentId = createResult.originalResponse.headers.find(_.name == RestResource.DeploymentHeader).map(_.value)
327	329	deploymentId shouldBe defined
328	330
329	331	Then("the deployment gets created")
330	332	WaitTestSupport.validFor("deployment visible", 5.second)(marathon.listDeploymentsForBaseGroup().value.size == 1)
331	333
332	334	When("the deployment is rolled back")
333	335	val deleteResult = marathon.deleteDeployment(deploymentId.get)
334	336	deleteResult should be(OK)
	337	val deleteId = deleteResult.originalResponse.headers.find(_.name == RestResource.DeploymentHeader).map(_.value)
	338	deleteId shouldBe defined
335	339
336	340	Then("the deployment should be gone")
337		waitForEvent("deployment_failed") // ScalePod
338		waitForDeployment(deleteResult) // StopPod
	341	val waitingFor = mutable.Map[String, CallbackEvent => Boolean](
	342	"deployment_failed" -> (_.id == deploymentId.value), // StartPod
	343	"deployment_success" -> (_.id == deleteId.value) // StopPod
	344	)
	345	waitForEventsWith(s"waiting for canceled ${deploymentId.value} and successful ${deleteId.value}", waitingFor)
	346
339	347	WaitTestSupport.waitUntil("Deployments get removed from the queue") {
340	348	marathon.listDeploymentsForBaseGroup().value.isEmpty
341	349	}
342	350
343	351	Then("the pod should also be gone")
344	352	marathon.pod(pod.id) should be(NotFound)
345	353	}
346	354
Show All 30 Lines

View Options

src/test/scala/mesosphere/marathon/integration/setup/MarathonTest.scala

Show First 20 Lines • Show All 517 Lines • ▼ Show 20 Line(s)
518	518		}
519	519		} catch {
520	520		case _: NoSuchElementException =>
521	521		None
522	522		case _: TestFailedDueToTimeoutException =>
523	523		None
524	524		}
525	525
	526		/**
	527		* Method waits for events and calls their callbacks independently of the events order. It receives a
		ichernetskyUnsubmitted Done `it's` => `their`.
	528		* mutable map of EventId -> Callback e.g. mutable.Map("deployment_failed" -> _.id == deploymentId),
	529		* checks every event for it's existence in the map and if found, calls it's callback method. If successful, the entry
	530		* is removed from the map. Returns if the map is empty.
	531		*/
	532		def waitForEventsWith(
	533		description: String,
	534		waitingFor: mutable.Map[String, CallbackEvent => Boolean],
		jdefUnsubmitted Not Done this func doesn't actually need to accept a mutable Map. The parameter could easily be an immutable map. If a mutable structure is really needed, it then this func should convert the Map param from immutable to mutable. Exposing an API that accepts mutable things is something that we should strongly discourage except in very, very rare circumstances. IMO there is not justification here for doing so.
		jdefUnsubmitted Not Done JIRA MARATHON-7655 Address review feedback for D935 integration test changes
		zen-dogAuthorUnsubmitted Not Done I thought about taking in an `immutable.Map` and converting it to `mutable` but it's a internal tester helper and I decided it's not worth extra effort.
	535		maxWait: FiniteDuration = patienceConfig.timeout.toMillis.millis) = {
		jdefUnsubmitted Not Done nit: public func, should have a return type
	536		waitForEventMatching(description) { event =>
	537		if (waitingFor.get(event.eventType).fold(false)(fn => fn(event))) {
	538		waitingFor -= event.eventType
	539		}
	540		waitingFor.isEmpty
	541		}
	542		}
	543
526	544		def waitForEventMatching(
527	545		description: String,
528	546		maxWait: FiniteDuration = patienceConfig.timeout.toMillis.millis)(fn: CallbackEvent => Boolean): CallbackEvent = {
529	547		val deadline = maxWait.fromNow
530	548		@tailrec
531	549		def iter(): CallbackEvent = {
532	550		nextEvent(deadline) match {
533	551		case Some(ITConnected) =>
▲ Show 20 Lines • Show All 303 Lines • Show Last 20 Lines

Fix flaky MesosAppIntegrationTestClosedAll UsersActions

Details

Diff Detail

(๑′°︿°๑)

(๑′°︿°๑)

(๑′°︿°๑)

(๑′°︿°๑)

(๑′°︿°๑)

(๑′°︿°๑)

(๑′°︿°๑)

＼\ ٩( ᐛ )و /／

Revision Contents

Diff 3845

src/main/scala/mesosphere/marathon/core/group/impl/GroupManagerImpl.scala

src/test/scala/mesosphere/marathon/integration/AppDeployIntegrationTest.scala

src/test/scala/mesosphere/marathon/integration/MesosAppIntegrationTest.scala

src/test/scala/mesosphere/marathon/integration/setup/MarathonTest.scala

Fix flaky MesosAppIntegrationTest
ClosedAll Users
Actions