HomeMesosphereNo notifications. 4 unresolved issues.
  • Search
Differential D655

unit tests (and fixes) for networking migration
ClosedAll Users
Actions

Authored by jdef on Mar 28 2017, 10:37 PM.

Details

Reviewers
aquamatthias
timcharper
jenkins
• jasongilanfarr
Commits
rMARATHON2d31b4f83926: unit tests (and fixes) for networking migration
rMARATHON8f435300e719: unit tests (and fixes) for networking migration
rMARATHON11537361898d: unit tests (and fixes) for networking migration
rMARATHON2646e7643e5f: unit tests (and fixes) for networking migration
rMARATHON2b81cb4bfecd: unit tests (and fixes) for networking migration
rMARATHON1b0cd4c2f6cb: unit tests (and fixes) for networking migration
rMARATHON6d83c343b2f7: unit tests (and fixes) for networking migration
JIRA Issues
JIRA MARATHON-7032 Unit testing needed for app migration added for D201 network overhaul
Summary
  • additional unit tests for networking migration
  • some additional protobuf helpers
  • fixes for app/container conversion
Test Plan

sbt test

Diff Detail

Repository
rMARATHON marathon
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
There are a very large number of changes, so older changes are hidden. Show Older Changes
jdef planned changes to this revision.Mar 28 2017, 10:37 PM

Still some additional tests to write, pushing this up as a PREVIEW

jdef added reviewers: • jasongilanfarr, aquamatthias, timcharper.Mar 28 2017, 10:38 PM
jdef retitled this revision from unit tests (and fixes) for networking migration to PREVIEW: unit tests (and fixes) for networking migration.
jdef updated this revision to Diff 2424.Mar 28 2017, 11:10 PM

rebase to master

jenkins requested changes to this revision.Mar 28 2017, 11:20 PM
This revision now requires changes to proceed.Mar 28 2017, 11:20 PM
jdef planned changes to this revision.Mar 28 2017, 11:39 PM

Not sure what Jenkins is complaining about. But I've got more changes coming anyway...

jdef updated this revision to Diff 2429.Mar 29 2017, 3:30 AM
jdef retitled this revision from PREVIEW: unit tests (and fixes) for networking migration to unit tests (and fixes) for networking migration.Mar 29 2017, 3:31 AM
jenkins requested changes to this revision.Mar 29 2017, 3:31 AM
This revision now requires changes to proceed.Mar 29 2017, 3:31 AM
jenkins accepted this revision.Mar 29 2017, 4:01 AM

Ship it!

This revision is now accepted and ready to land.Mar 29 2017, 4:01 AM
aquamatthias requested changes to this revision.Mar 29 2017, 12:49 PM

I imported a state that is runnable with 1.4, applied your patch to master and started this version.
The migration failed with:

Caused by: mesosphere.marathon.ValidationFailedException: Validation failed: Failure(Set(GroupViolation(Set(MON_GROUP, framework-id, SERVERPORT, FORWARDHOST, MON_APP, id, MON_CONTACT, SERVICENAME, MON_PIPE_ID, dns-proxy-pool, FORWARDPORT, EXTADDR, transparent, EXTNETNAME, ETCDENDPOINT, IO, ext-dns-server, LISTENPORT),contains elements, which are not valid.,Some(env),Set(GroupViolation(ext-dns-server,not valid,Some((15)),Set(RuleViolation(ext-dns-server,must contain only alphanumeric chars or underscore, and must not begin with a number,Some(value)))), GroupViolation(dns-proxy-pool,not valid,Some((12)),Set(RuleViolation(dns-proxy-pool,must contain only alphanumeric chars or underscore, and must not begin with a number,Some(value)))), GroupViolation(framework-id,not valid,Some((6)),Set(RuleViolation(framework-id,must contain only alphanumeric chars or underscore, and must not begin with a number,Some(value))))))))

You can do the same by

src/main/scala/mesosphere/marathon/raml/AppConversion.scala
351

If this is specific to proto handling, should this code be part of AppDefinition.fromProto?

src/main/scala/mesosphere/marathon/raml/ContainerConversion.scala
199

The idea with obsolete was: it is only used in Migration code. Why do we need this here?

This revision now requires changes to proceed.Mar 29 2017, 12:49 PM
jdef marked 2 inline comments as done.Mar 29 2017, 6:45 PM
jdef added inline comments.
src/main/scala/mesosphere/marathon/raml/AppConversion.scala
351

no because AppDefinition.fromProto converts from protobuf-to-model. This code converts from protobuf-to-RAML -- different semantics.

src/main/scala/mesosphere/marathon/raml/ContainerConversion.scala
199

because the migration code leverages protobuf-to-RAML conversions for the migration. I added the conversion funcs to our existing xxxConversion classes.

jdef marked 2 inline comments as done.Mar 29 2017, 10:19 PM

looks like the app has environment variable names that contain hyphens. how did we ever get here?

jdef added a comment.Mar 29 2017, 10:38 PM

D201 introduced stronger environment variable name validation. model validation is pretty lax for envvar names: https://github.com/mesosphere/marathon/blob/releases/1.4/src/main/scala/mesosphere/marathon/state/EnvVarValue.scala#L56

jdef added a comment.Mar 30 2017, 12:23 AM
[info] Caused by: mesosphere.marathon.ValidationFailedException: Validation failed: Failure(Set(RuleViolation(List(hostname, CLUSTER),Missing value,Some(constraints))))
[info]  at mesosphere.marathon.api.v2.Validation$class.validateOrThrow(Validation.scala:23)
[info]  at mesosphere.marathon.api.v2.Validation$.validateOrThrow(Validation.scala:304)
[info]  at mesosphere.marathon.api.v2.AppsResource$$anonfun$appNormalization$1.apply(AppsResource.scala:409)
[info]  at mesosphere.marathon.api.v2.AppsResource$$anonfun$appNormalization$1.apply(AppsResource.scala:406)
[info]  at mesosphere.marathon.Normalization$$anon$1.normalized(Normalization.scala:15)
[info]  at mesosphere.marathon.Normalization$Normalized$.normalize$extension(Normalization.scala:11)
[info]  at mesosphere.marathon.storage.migration.MigrationTo1_5$$anonfun$migrateServiceFlow$1.apply(MigrationTo1_5.scala:113)
[info]  at mesosphere.marathon.storage.migration.MigrationTo1_5$$anonfun$migrateServiceFlow$1.apply(MigrationTo1_5.scala:110)
jdef planned changes to this revision.Mar 30 2017, 12:23 AM
jdef updated this revision to Diff 2463.Mar 31 2017, 5:50 PM
  • rMARATHON0a64b1f84c14 sanity check: precise unit tests for envvar conversion, relocate envvar proto-to-RAML conversion
  • rMARATHON08a79d58d722 app envvar names should not use strict name validation to avoid API compat breakage
  • rebase to master
jdef added a comment.Mar 31 2017, 5:52 PM

@aquamatthias your concerns have been addressed. migration of the exported ZK blob now completes successfully

jenkins requested changes to this revision.Mar 31 2017, 5:55 PM
This revision now requires changes to proceed.Mar 31 2017, 5:55 PM
jenkins accepted this revision.Mar 31 2017, 6:27 PM

Ship it!

• jasongilanfarr accepted this revision.Mar 31 2017, 6:31 PM
• jasongilanfarr added inline comments.
src/main/scala/mesosphere/marathon/raml/AppConversion.scala
351

Should we have a Boolean to indicate whether it's empty or unset?

355

all the more reason to move to Json in storage soon

src/main/scala/mesosphere/marathon/storage/migration/MigrationTo1_5.scala
50

Should we validate too?

jdef added inline comments.Mar 31 2017, 6:44 PM
src/main/scala/mesosphere/marathon/raml/AppConversion.scala
351

there's no need. the state model doesn't define ports (definitions or mappings) wrapped in Option. RAML does, and that's where the complication comes in

src/main/scala/mesosphere/marathon/storage/migration/MigrationTo1_5.scala
50

validation is baked into the AppsResource normalizer; no additional steps needed here

aquamatthias requested changes to this revision.Apr 3 2017, 10:58 AM

I applied the migration to the production zk state. The migration now runs successfully - great!

Have you added a test case for all scenarios that failed before?
So we can make sure, we do not run again into this situation?

During the migration, laxer validations are applied.
This leads to a situation, where no change can be applied after the migration, since the validation on the root group fails with:

{
    "details": [
        {
            "errors": [
                "Missing value"
            ],
            "path": "/groups(0)/groups(2)/apps(37)/constraints"
        },
        {
            "errors": [
                "Missing value"
            ],
            "path": "/groups(0)/groups(2)/apps(4)/constraints"
        },
        {
            "errors": [
                "Missing value"
            ],
            "path": "/groups(0)/groups(2)/apps(36)/constraints"
        },
        {
            "errors": [
                "Missing value"
            ],
            "path": "/groups(0)/groups(2)/apps(0)/constraints"
        }
    ],
    "message": "Object is not valid"
}
aquamatthias added inline comments.Apr 3 2017, 11:17 AM
src/main/scala/mesosphere/marathon/api/v2/validation/EnvVarValidation.scala
23

This turns env var name validation off for apps.
I tried to add an env var like this:

"env": {
  "&&$G$ ^^^ kjshg dfg dfjkhgdfk g! doo": "marathon"
}

which should not have been accepted, but passes validation.

src/main/scala/mesosphere/marathon/raml/ContainerConversion.scala
199

Any suggestion to separate the migration path from the standard code path?
Or is it simply not obsolete?

src/main/scala/mesosphere/marathon/storage/migration/MigrationTo1_5.scala
85

Please add a short explanation, why current root must be last.

jdef planned changes to this revision.Apr 3 2017, 3:11 PM
jdef marked an inline comment as done.

suspect that constraint validation is failing because the recent constraint "fixes" that landed on master should have removed the largely redundant (but incorrect) validation func in AppDefinition. will create a separate diff for that and rebase this once that lands.

src/main/scala/mesosphere/marathon/api/v2/validation/EnvVarValidation.scala
23

D201 added a regression that invalidated names that were considered valid prior to D201. changes here fix that.

src/main/scala/mesosphere/marathon/raml/ContainerConversion.scala
199

I thought it would be pretty consistent to keep all the RAML conversion code in the same package. I could see, perhaps, segregating the protobuf-to-RAML code into separate conversion classes .. but then I think that complicates maintenance efforts - from a maintenance perspective I'd rather have everything in the same file.

do you have any suggestions?

we name proto fields OBSOLETE when they've been deprecated and no longer used for new items written to storage. according to that rule, I've named the fields appropriately

jdef updated this revision to Diff 2477.Apr 3 2017, 6:43 PM
jdef marked an inline comment as done.
jenkins requested changes to this revision.Apr 3 2017, 6:43 PM
This revision now requires changes to proceed.Apr 3 2017, 6:43 PM
jdef updated this revision to Diff 2478.Apr 3 2017, 6:47 PM

rebase to master

jenkins requested changes to this revision.Apr 3 2017, 6:54 PM
This revision now requires changes to proceed.Apr 3 2017, 6:54 PM
jenkins accepted this revision.Apr 3 2017, 7:14 PM

Thanks.

jenkins requested changes to this revision.Apr 3 2017, 8:50 PM
jenkins accepted this revision.Apr 3 2017, 9:55 PM

Ship it!

aquamatthias accepted this revision.Apr 4 2017, 6:03 PM

Thanks!

src/test/scala/mesosphere/mesos/protos/ImplicitsSpec.scala
6

Nit: please use double package notation. You should not need this import then.

This revision is now accepted and ready to land.Apr 4 2017, 6:03 PM
jdef added inline comments.Apr 4 2017, 7:07 PM
src/test/scala/mesosphere/mesos/protos/ImplicitsSpec.scala
6

double package only applies for mesosphere.marathon packages

Closed by commit rMARATHON2b81cb4bfecd: unit tests (and fixes) for networking migration (authored by jdef). · Explain WhyApr 4 2017, 7:07 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathPackages
Mdocs/docs/rest-api/public/api/v2/types/app.raml (2 lines)
Mdocs/docs/rest-api/public/api/v2/types/environmentVariable.raml (7 lines)
Msrc/main/scala/mesosphere/marathon/api/v2/AppNormalization.scala (13 lines)
Msrc/main/scala/mesosphere/marathon/api/v2/validation/AppValidation.scala (4 lines)
Msrc/main/scala/mesosphere/marathon/api/v2/validation/EnvVarValidation.scala (11 lines)
Msrc/main/scala/mesosphere/marathon/api/v2/validation/PodsValidation.scala (4 lines)
Msrc/main/scala/mesosphere/marathon/raml/AppConversion.scala (34 lines)
Msrc/main/scala/mesosphere/marathon/raml/ContainerConversion.scala (6 lines)
Msrc/main/scala/mesosphere/marathon/raml/EnvVarConversion.scala (14 lines)
Msrc/main/scala/mesosphere/marathon/raml/Environment.scala (15 lines)
Msrc/main/scala/mesosphere/marathon/raml/NetworkConversion.scala (11 lines)
Msrc/main/scala/mesosphere/marathon/storage/migration/MigrationTo1_5.scala (124 lines)
Msrc/main/scala/mesosphere/mesos/protos/Range.scala (4 lines)
Msrc/main/scala/mesosphere/mesos/protos/RangesResource.scala (7 lines)
Msrc/main/scala/mesosphere/mesos/protos/ScalarResource.scala (7 lines)
Msrc/test/scala/mesosphere/marathon/api/v2/AppsResourceTest.scala (6 lines)
Msrc/test/scala/mesosphere/marathon/api/v2/json/AppDefinitionFormatsTest.scala (2 lines)
Msrc/test/scala/mesosphere/marathon/api/v2/validation/EnvVarValidationTest.scala (40 lines)
Msrc/test/scala/mesosphere/marathon/raml/AppConversionTest.scala (2 lines)
Msrc/test/scala/mesosphere/marathon/raml/ContainerConversionTest.scala (2 lines)
AMsrc/test/scala/mesosphere/marathon/raml/EnvVarConversionTest.scala (73 lines)
AMsrc/test/scala/mesosphere/marathon/storage/migration/MigrationTo1_5Test.scala (420 lines)
Msrc/test/scala/mesosphere/mesos/protos/ImplicitsSpec.scala (2 lines)

Diff 2502

View Options

docs/docs/rest-api/public/api/v2/types/app.raml

Show First 20 Lines • Show All 164 Lines • ▼ Show 20 Line(s)
165165 type: number
166166 default: 0
167167 description: |
168168 How much disk space is needed for this application.
169169 This number does not have to be an integer, but can be a fraction.
170170 format: double
171171 minimum: 0
172172 env?:
173 type: env.EnvVars
173 type: env.LegacyEnvVars
174174 (pragma.omitEmpty):
175175 executor?:
176176 type: string
177177 pattern: ^(//cmd)|(/?[^/]+(/[^/]+)*)|$
178178 default: ""
179179 description: |
180180 The executor to use to launch this application.
181181 Different executors are available.
▲ Show 20 Lines • Show All 142 Lines • Show Last 20 Lines
View Options

docs/docs/rest-api/public/api/v2/types/environmentVariable.raml

Show All 12 Lines
1313 description: |
1414 The name of the secret to refer to. At runtime, the value of the
1515 secret will be injected into the value of the variable.
1616 EnvVarValueOrSecret: (EnvVarValue | EnvVarSecretRef)
1717 EnvVars:
1818 type: object
1919 properties:
2020 /^[a-zA-Z_][a-zA-Z0-9_]*$/: EnvVarValueOrSecret
21 LegacyEnvVars:
22 type: object
23 usage: |
24 The original v2 apps API did not place any restrictions on environment variable names.
25 New APIs should use EnvVars whenever possible.
26 properties:
27 /^.*$/: EnvVarValueOrSecret
View Options

src/main/scala/mesosphere/marathon/api/v2/AppNormalization.scala

Show First 20 Lines • Show All 283 Lines • ▼ Show 20 Line(s)
284284 readinessChecks = app.readinessChecks.map(normalizeReadinessCheck)
285285 )
286286 }
287287
288288 case class Networks(config: Config, networks: Option[Seq[Network]])
289289
290290 object Networks {
291291 implicit val normalizedNetworks: Normalization[Networks] = Normalization { n =>
292 n.config.defaultNetworkName.map { _ =>
293 n.copy(networks = n.networks.map(_.map {
292 // IMPORTANT: only evaluate config.defaultNetworkName if we actually need it
293 n.copy(networks = n.networks.map{ networks =>
294 networks.map {
294295 case x: Network if x.name.isEmpty && x.mode == NetworkMode.Container => x.copy(name = n.config.defaultNetworkName)
295296 case x => x
296 }))
297 }.getOrElse(n)
298297 }
298 })
299299 }
300 }
300301
301302 def apply(config: Config): Normalization[App] = Normalization { app =>
302303 val networks = Networks(config, Some(app.networks)).normalize.networks.filter(_.nonEmpty).getOrElse(DefaultNetworks)
303304 val container = NetworkedContainer(Some(networks), app.container).normalize.container
304305
305306 val defaultUnreachable: UnreachableStrategy = {
306307 val hasPersistentVols = app.container.exists(_.volumes.exists(_.persistent.nonEmpty))
307308 state.UnreachableStrategy.default(hasPersistentVols).toRaml
308309 }
309310
310311 // requirePorts only applies for host-mode networking
311312 val requirePorts = networks.find(_.mode != NetworkMode.Host).fold(app.requirePorts)(_ => false)
312313
313314 app.copy(
314315 container = container,
315316 networks = networks,
316317 unreachableStrategy = app.unreachableStrategy.orElse(Option(defaultUnreachable)),
317318 requirePorts = requirePorts
318319 )
319320 }
320321
321 /** dynamic app normalization configuration, useful for testing */
322 /** dynamic app normalization configuration, useful for migration and/or testing */
322323 trait Config {
323324 def defaultNetworkName: Option[String]
324325 }
325326
326327 /** static app normalization configuration */
327328 case class Configure(override val defaultNetworkName: Option[String]) extends Config
328329
329330 /**
▲ Show 20 Lines • Show All 58 Lines • ▼ Show 20 Line(s)
388389 id = update.id.map(id => PathId(id).canonicalPath(base).toString),
389390 dependencies = update.dependencies.map(_.map(dep => PathId(dep).canonicalPath(base).toString))
390391 ).asInstanceOf[T]
391392 case app: App =>
392393 app.copy(
393394 id = PathId(app.id).canonicalPath(base).toString,
394395 dependencies = app.dependencies.map(dep => PathId(dep).canonicalPath(base).toString)
395396 ).asInstanceOf[T]
396 case _ => throw new SerializationFailedException("withCanonizedIds only applies for App and AppUpdate")
397 case _ => throw SerializationFailedException("withCanonizedIds only applies for App and AppUpdate")
397398 }
398399}
View Options

src/main/scala/mesosphere/marathon/api/v2/validation/AppValidation.scala

Show First 20 Lines • Show All 277 Lines • ▼ Show 20 Line(s)
278278 !(update.container.exists(_.docker.exists(_.network.nonEmpty)) && update.networks.nonEmpty)
279279 }
280280 )
281281
282282 def validateCanonicalAppUpdateAPI(enabledFeatures: Set[String]): Validator[AppUpdate] = forAll(
283283 validator[AppUpdate] { update =>
284284 update.id.map(PathId(_)) as "id" is optional(valid)
285285 update.dependencies.map(_.map(PathId(_))) as "dependencies" is optional(every(valid))
286 update.env is optional(envValidator(update.secrets.getOrElse(Map.empty), enabledFeatures))
286 update.env is optional(envValidator(strictNameValidation = false, update.secrets.getOrElse(Map.empty), enabledFeatures))
287287 update.secrets is optional({ secrets: Map[String, SecretDef] =>
288288 secrets.nonEmpty
289289 } -> (featureEnabled(enabledFeatures, Features.SECRETS)))
290290 update.secrets is optional(featureEnabledImplies(enabledFeatures, Features.SECRETS)(every(secretEntryValidator)))
291291 update.storeUrls is optional(every(urlIsValid))
292292 update.fetch is optional(every(valid))
293293 update.portDefinitions is optional(portDefinitionsValidator)
294294 update.container is optional(valid(validContainer(enabledFeatures, update.networks.getOrElse(Nil))))
▲ Show 20 Lines • Show All 85 Lines • ▼ Show 20 Line(s)
380380 app is containsCmdArgsOrContainer
381381 app.healthChecks is every(portIndexIsValid(portIndices(app)))
382382 app must haveAtMostOneMesosHealthCheck
383383 app.fetch is every(valid)
384384 app.secrets is valid({ secrets: Map[String, SecretDef] =>
385385 secrets.nonEmpty
386386 } -> (featureEnabled(enabledFeatures, Features.SECRETS)))
387387 app.secrets is valid(featureEnabledImplies(enabledFeatures, Features.SECRETS)(every(secretEntryValidator)))
388 app.env is envValidator(app.secrets, enabledFeatures)
388 app.env is envValidator(strictNameValidation = false, app.secrets, enabledFeatures)
389389 app.acceptedResourceRoles is valid(optional(ResourceRole.validAcceptedResourceRoles(app.residency.isDefined) and notEmpty))
390390 app must complyWithGpuRules(enabledFeatures)
391391 app must complyWithMigrationAPI
392392 app must complyWithReadinessCheckRules
393393 app must complyWithResidencyRules
394394 app must complyWithSingleInstanceLabelRules
395395 app must complyWithUpgradeStrategyRules
396396 app must complyWithDockerNetworkingRules
▲ Show 20 Lines • Show All 107 Lines • Show Last 20 Lines
View Options

src/main/scala/mesosphere/marathon/api/v2/validation/EnvVarValidation.scala

11package mesosphere.marathon
22package api.v2.validation
33
44import com.wix.accord._
55import com.wix.accord.dsl._
66import mesosphere.marathon.raml.{ EnvVarSecretRef, EnvVarValueOrSecret, SecretDef }
77
88/**
99 * RAML-generated validation doesn't cover environment variable names yet
1010 */
1111trait EnvVarValidation {
1212 import mesosphere.marathon.api.v2.Validation._
1313 import EnvVarValidationMessages._
1414
1515 val EnvVarNamePattern = """^[a-zA-Z_][a-zA-Z0-9_]*$""".r
1616
17 val validEnvVar: Validator[(String, EnvVarValueOrSecret)] = {
17 /**
18 * @param strictNameValidation corresponds to RAML type EnvVars when true and LegacyEnvVars when false.
19 */
20 def validEnvVar(strictNameValidation: Boolean): Validator[(String, EnvVarValueOrSecret)] = {
1821
1922 val validName = validator[String] { name =>
20 name should matchRegexWithFailureMessage(EnvVarNamePattern, MustContainOnlyAlphanumeric)
23 name is implied(strictNameValidation)(matchRegexWithFailureMessage(EnvVarNamePattern, MustContainOnlyAlphanumeric))
aquamatthiasUnsubmitted
Done

This turns env var name validation off for apps.
I tried to add an env var like this:

"env": {
  "&&$G$ ^^^ kjshg dfg dfjkhgdfk g! doo": "marathon"
}

which should not have been accepted, but passes validation.

jdefAuthorUnsubmitted
Not Done

D201 added a regression that invalidated names that were considered valid prior to D201. changes here fix that.

2124 name is notEmpty
2225 } and isTrue[String](VariableNameTooLong) { _.length < 255 }
2326
2427 validator[(String, EnvVarValueOrSecret)] { t =>
2528 // use of "value" relies on special behavior in Validation that humanizes generated error messages
2629 t._1 as "value" is validName
2730 }
2831 }
2932
30 def envValidator(secrets: Map[String, SecretDef], enabledFeatures: Set[String]) = forAll(
33 def envValidator(strictNameValidation: Boolean, secrets: Map[String, SecretDef], enabledFeatures: Set[String]) = forAll(
3134 validator[Map[String, EnvVarValueOrSecret]] { env =>
32 env is every(validEnvVar)
35 env is every(validEnvVar(strictNameValidation))
3336 },
3437 isTrue(UseOfSecretRefsRequiresSecretFeature) { (env: Map[String, EnvVarValueOrSecret]) =>
3538 // if the secrets feature is not enabled then don't allow EnvVarSecretRef's in the environment
3639 if (!enabledFeatures.contains(Features.SECRETS))
3740 env.values.count {
3841 case _: EnvVarSecretRef => true
3942 case _ => false
4043 } == 0
Show All 13 Lines
View Options

src/main/scala/mesosphere/marathon/api/v2/validation/PodsValidation.scala

Show First 20 Lines • Show All 129 Lines • ▼ Show 20 Line(s)
130130 lc.killGracePeriodSeconds.getOrElse(0.0) should be > 0.0
131131 }
132132
133133 def containerValidator(pod: Pod, enabledFeatures: Set[String], mesosMasterVersion: SemanticVersion): Validator[PodContainer] =
134134 validator[PodContainer] { container =>
135135 container.resources is valid(resourceValidator)
136136 container.endpoints is every(endpointValidator(pod.networks))
137137 container.image.getOrElse(Image(ImageType.Docker, "abc")) is valid(imageValidator)
138 container.environment is envValidator(pod.secrets, enabledFeatures)
138 container.environment is envValidator(strictNameValidation = true, pod.secrets, enabledFeatures)
139139 container.healthCheck is optional(healthCheckValidator(container.endpoints, mesosMasterVersion))
140140 container.volumeMounts is every(volumeMountValidator(pod.volumes))
141141 container.artifacts is every(artifactValidator)
142142 }
143143
144144 def volumeValidator(containers: Seq[PodContainer]): Validator[Volume] = validator[Volume] { volume =>
145145 volume.host is optional(notEmpty)
146146 } and isTrue[Volume]("volume must be referenced by at least one container") { v =>
Show All 24 Lines
171171 val endpointHostPortsUnique: Validator[Pod] = isTrue("Host ports are unique") { pod: Pod =>
172172 val hostPorts = pod.containers.flatMap(_.endpoints.flatMap(_.hostPort)).filter(_ != 0)
173173 hostPorts.distinct.size == hostPorts.size
174174 }
175175
176176 def podDefValidator(enabledFeatures: Set[String], mesosMasterVersion: SemanticVersion): Validator[Pod] = validator[Pod] { pod =>
177177 PathId(pod.id) as "id" is valid and PathId.absolutePathValidator and PathId.nonEmptyPath
178178 pod.user is optional(notEmpty)
179 pod.environment is envValidator(pod.secrets, enabledFeatures)
179 pod.environment is envValidator(strictNameValidation = true, pod.secrets, enabledFeatures)
180180 pod.volumes is every(volumeValidator(pod.containers)) and isTrue("volume names are unique") { volumes: Seq[Volume] =>
181181 val names = volumes.map(_.name)
182182 names.distinct.size == names.size
183183 }
184184 pod.containers is notEmpty and every(containerValidator(pod, enabledFeatures, mesosMasterVersion))
185185 pod.containers is isTrue("container names are unique") { containers: Seq[PodContainer] =>
186186 val names = pod.containers.map(_.name)
187187 names.distinct.size == names.size
Show All 13 Lines
View Options

src/main/scala/mesosphere/marathon/raml/AppConversion.scala

Show First 20 Lines • Show All 277 Lines • ▼ Show 20 Line(s)
278278 implicit val discoveryProtoRamlWriter: Writes[Protos.ObsoleteDiscoveryInfo, IpDiscovery] = Writes { di =>
279279 IpDiscovery(
280280 ports = di.whenOrElse(_.getPortsCount > 0, _.getPortsList.map(_.toRaml[IpDiscoveryPort])(collection.breakOut), IpDiscovery.DefaultPorts)
281281 )
282282 }
283283
284284 implicit val ipAddressProtoRamlWriter: Writes[Protos.ObsoleteIpAddress, IpAddress] = Writes { ip =>
285285 IpAddress(
286 discovery = ip.when(_.hasDiscoveryInfo, _.getDiscoveryInfo.toRaml).orElse(IpAddress.DefaultDiscovery),
286 discovery = ip.collect {
287 case x if x.hasDiscoveryInfo && x.getDiscoveryInfo.getPortsCount > 0 => x.getDiscoveryInfo.toRaml
288 }.orElse(IpAddress.DefaultDiscovery),
287289 groups = ip.whenOrElse(_.getGroupsCount > 0, _.getGroupsList.to[Set], IpAddress.DefaultGroups),
288290 labels = ip.whenOrElse(_.getLabelsCount > 0, _.getLabelsList.to[Seq].fromProto, IpAddress.DefaultLabels),
289291 networkName = ip.when(_.hasNetworkName, _.getNetworkName).orElse(IpAddress.DefaultNetworkName)
290292 )
291293 }
292294
293295 implicit val appProtoRamlWriter: Writes[Protos.ServiceDefinition, App] = Writes { service =>
294296 import mesosphere.mesos.protos.Resource
295297
296298 val resourcesMap: Map[String, Double] =
297299 service.getResourcesList.map {
298300 r => r.getName -> (r.getScalar.getValue: Double)
299301 }(collection.breakOut)
300302
301 def envMap: Map[String, EnvVarValueOrSecret] = (
302 if (service.hasCmd) {
303 service.getCmd.getEnvironment.getVariablesList.map { item =>
304 item.getName -> EnvVarValue(item.getValue)
305 }.toMap
306 } else {
307 App.DefaultEnv
308 }
309 ) ++ service.getEnvVarReferencesList.withFilter(_.getType == Protos.EnvVarReference.Type.SECRET).map { secretRef =>
310 secretRef.getName -> EnvVarSecretRef(secretRef.getSecretRef.getSecretId)
311 }
312
313303 val version = service.when(_.hasVersion, s => Timestamp(s.getVersion).toOffsetDateTime).orElse(App.DefaultVersion)
314304 val versionInfo: Option[VersionInfo] =
315305 if (service.hasLastScalingAt) Option(VersionInfo(
316306 lastConfigChangeAt = Timestamp(service.getLastConfigChangeAt).toOffsetDateTime,
317307 lastScalingAt = Timestamp(service.getLastScalingAt).toOffsetDateTime
318308 ))
319309 else None
320310
321 App(
311 val app = App(
322312 id = service.getId,
323313 acceptedResourceRoles = if (service.hasAcceptedResourceRoles && service.getAcceptedResourceRoles.getRoleCount > 0) Option(service.getAcceptedResourceRoles.getRoleList.to[Set]) else App.DefaultAcceptedResourceRoles,
324314 args = if (service.hasCmd && service.getCmd.getArgumentsCount > 0) service.getCmd.getArgumentsList.to[Seq] else App.DefaultArgs,
325315 backoffFactor = service.whenOrElse(_.hasBackoffFactor, _.getBackoffFactor, App.DefaultBackoffFactor),
326316 backoffSeconds = service.whenOrElse(_.hasBackoff, b => (b.getBackoff / 1000L).toInt, App.DefaultBackoffSeconds),
327317 cmd = if (service.hasCmd && service.getCmd.hasValue) Option(service.getCmd.getValue) else App.DefaultCmd,
328318 constraints = service.whenOrElse(_.getConstraintsCount > 0, _.getConstraintsList.map(_.toRaml[Seq[String]])(collection.breakOut), App.DefaultConstraints),
329319 container = service.when(_.hasContainer, _.getContainer.toRaml).orElse(App.DefaultContainer),
330320 cpus = resourcesMap.getOrElse(Resource.CPUS, App.DefaultCpus),
331321 dependencies = service.whenOrElse(_.getDependenciesCount > 0, _.getDependenciesList.to[Set], App.DefaultDependencies),
332322 disk = resourcesMap.getOrElse(Resource.DISK, App.DefaultDisk),
333 env = envMap,
323 env = service.whenOrElse(_.hasCmd, s => (s.getCmd.getEnvironment.getVariablesList.to[Seq], s.getEnvVarReferencesList.to[Seq]).toRaml, App.DefaultEnv),
334324 executor = service.whenOrElse(_.hasExecutor, _.getExecutor, App.DefaultExecutor),
335325 fetch = if (service.hasCmd && service.getCmd.getUrisCount > 0) service.getCmd.getUrisList.toRaml else App.DefaultFetch,
336326 healthChecks = service.whenOrElse(_.getHealthChecksCount > 0, _.getHealthChecksList.toRaml.to[Set], App.DefaultHealthChecks),
337327 instances = service.whenOrElse(_.hasInstances, _.getInstances, App.DefaultInstances),
338328 labels = service.getLabelsList.map { label => label.getKey -> label.getValue }(collection.breakOut),
339329 maxLaunchDelaySeconds = service.whenOrElse(_.hasMaxLaunchDelay, m => (m.getMaxLaunchDelay / 1000L).toInt, App.DefaultMaxLaunchDelaySeconds),
340330 mem = resourcesMap.getOrElse(Resource.MEM, App.DefaultMem),
341331 gpus = resourcesMap.get(Resource.GPUS).fold(App.DefaultGpus)(_.toInt),
Show All 11 Lines
353343 upgradeStrategy = service.when(_.hasUpgradeStrategy, _.getUpgradeStrategy.toRaml).orElse(App.DefaultUpgradeStrategy),
354344 uris = None, // not stored in protobuf
355345 user = if (service.hasCmd && service.getCmd.hasUser) Option(service.getCmd.getUser) else App.DefaultUser,
356346 version = version,
357347 versionInfo = versionInfo, // we restore this but App-to-AppDefinition conversion drops it...
358348 killSelection = service.whenOrElse(_.hasKillSelection, _.getKillSelection.toRaml, App.DefaultKillSelection),
359349 unreachableStrategy = service.when(_.hasUnreachableStrategy, _.getUnreachableStrategy.toRaml).orElse(App.DefaultUnreachableStrategy)
360350 )
351 // special ports normalization when converting from protobuf, because the protos don't allow us to distinguish
aquamatthiasUnsubmitted
Done

If this is specific to proto handling, should this code be part of AppDefinition.fromProto?

jdefAuthorUnsubmitted
Not Done

no because AppDefinition.fromProto converts from protobuf-to-model. This code converts from protobuf-to-RAML -- different semantics.

jasongilanfarrUnsubmitted
Not Done

Should we have a Boolean to indicate whether it's empty or unset?

jdefAuthorUnsubmitted
Not Done

there's no need. the state model doesn't define ports (definitions or mappings) wrapped in Option. RAML does, and that's where the complication comes in

352 // between "I specified an empty set of ports" and "I specified a null set of ports" (for definitions and mappings).
353 // note that we don't clear app.container.docker.portMappings because those may be valid in some cases: some other
354 // normalization code should deal with that.
355 (app.container, app.ipAddress) match {
jasongilanfarrUnsubmitted
Not Done

all the more reason to move to Json in storage soon

356 case (ct, Some(ip)) if ct.exists(_.`type` != EngineType.Mesos) || ip.discovery.exists(_.ports.nonEmpty) =>
357 app.copy(
358 container = ct.map(_.copy(portMappings = None)),
359 portDefinitions = None,
360 requirePorts = false
361 )
362 case _ =>
363 app
364 }
361365 }
362366
363367 /**
364368 * @return an app update generated from an app, but without the `version` field (since that can only
365369 * be combined with `id` for app updates)
366370 */
367371 implicit val appToUpdate: Writes[App, AppUpdate] = Writes { app =>
368372 AppUpdate(
▲ Show 20 Lines • Show All 65 Lines • Show Last 20 Lines
View Options

src/main/scala/mesosphere/marathon/raml/ContainerConversion.scala

Show First 20 Lines • Show All 189 Lines • ▼ Show 20 Line(s)
190190 `type` = container.when(_.hasType, _.getType.toRaml).getOrElse(Container.DefaultType),
191191 docker =
192192 container.collect {
193193 case x if x.hasDocker => x.getDocker.toRaml
194194 case x if x.hasMesosDocker => x.getMesosDocker.toRaml
195195 }.orElse(Container.DefaultDocker),
196196 appc = container.when(_.hasMesosAppC, _.getMesosAppC.toRaml).orElse(Container.DefaultAppc),
197197 volumes = container.whenOrElse(_.getVolumesCount > 0, _.getVolumesList.map(_.toRaml)(collection.breakOut), Container.DefaultVolumes),
198 portMappings = Option(Seq.empty[ContainerPortMapping]).unless( // RAML default is None, which is not what we want
199 container.when(_.getPortMappingsCount > 0, _.getPortMappingsList.map(_.toRaml)(collection.breakOut)))
198 portMappings = container.collect {
199 case x if !x.hasDocker || x.getDocker.getOBSOLETEPortMappingsCount == 0 =>
aquamatthiasUnsubmitted
Done

The idea with obsolete was: it is only used in Migration code. Why do we need this here?

jdefAuthorUnsubmitted
Not Done

because the migration code leverages protobuf-to-RAML conversions for the migration. I added the conversion funcs to our existing xxxConversion classes.

aquamatthiasUnsubmitted
Not Done

Any suggestion to separate the migration path from the standard code path?
Or is it simply not obsolete?

jdefAuthorUnsubmitted
Not Done

I thought it would be pretty consistent to keep all the RAML conversion code in the same package. I could see, perhaps, segregating the protobuf-to-RAML code into separate conversion classes .. but then I think that complicates maintenance efforts - from a maintenance perspective I'd rather have everything in the same file.

do you have any suggestions?

we name proto fields OBSOLETE when they've been deprecated and no longer used for new items written to storage. according to that rule, I've named the fields appropriately

200 container.getPortMappingsList.map(_.toRaml)(collection.breakOut)
201 }
200202 )
201203 }
202204}
203205
204206object ContainerConversion extends ContainerConversion
View Options

src/main/scala/mesosphere/marathon/raml/EnvVarConversion.scala

11package mesosphere.marathon
22package raml
33
4import mesosphere.marathon.state
4import org.apache.mesos.{ Protos => Mesos }
55
66import scala.collection.immutable.Map
77
88trait EnvVarConversion {
99 implicit val envVarRamlWrites: Writes[Map[String, state.EnvVarValue], Map[String, EnvVarValueOrSecret]] =
1010 Writes {
1111 _.map {
1212 case (name, state.EnvVarString(v)) => name -> EnvVarValue(v)
1313 case (name, state.EnvVarSecretRef(v)) => name -> EnvVarSecretRef(v)
1414 }
1515 }
1616
1717 implicit val envVarReads: Reads[Map[String, EnvVarValueOrSecret], Map[String, state.EnvVarValue]] =
1818 Reads {
1919 _.map {
2020 case (name, EnvVarValue(v)) => name -> state.EnvVarString(v)
2121 case (name, EnvVarSecretRef(v)) => name -> state.EnvVarSecretRef(v)
22 }
23 }
24
25 implicit val envProtoRamlWrites: Writes[(Seq[Mesos.Environment.Variable], Seq[Protos.EnvVarReference]), Map[String, EnvVarValueOrSecret]] =
26 Writes {
27 case (env, refs) =>
28 val vanillaEnv: Map[String, EnvVarValueOrSecret] = env.map { item =>
29 item.getName -> EnvVarValue(item.getValue)
30 }(collection.breakOut)
31
32 vanillaEnv ++ refs.withFilter(_.getType == Protos.EnvVarReference.Type.SECRET).map { secretRef =>
33 secretRef.getName -> EnvVarSecretRef(secretRef.getSecretRef.getSecretId)
2234 }
2335 }
2436}
2537
2638object EnvVarConversion extends EnvVarConversion
View Options

src/main/scala/mesosphere/marathon/raml/Environment.scala

11package mesosphere.marathon
22package raml
33
44/**
55 * Helpers for quick environment variable construction
66 */
77object Environment {
88 def apply(kv: (String, String)*): Map[String, EnvVarValueOrSecret] =
99 kv.map { case (k, v) => k -> EnvVarValue(v) }(collection.breakOut)
1010 def apply(env: Map[String, String]): Map[String, EnvVarValueOrSecret] = env.mapValues(EnvVarValue(_))
11
12 object Implicits {
13 implicit class WithSecrets(val env: Map[String, EnvVarValueOrSecret]) extends AnyVal {
14 def withSecrets(nameToSecretRef: (String, String)*): Map[String, EnvVarValueOrSecret] = {
15 env ++ nameToSecretRef.map {
16 case (name, secretRef) =>
17 name -> EnvVarSecretRef(secretRef)
18 }
19 }
20
21 def withSecrets(namesToSecretRefs: Map[String, String]): Map[String, EnvVarValueOrSecret] = {
22 env ++ namesToSecretRefs.mapValues(EnvVarSecretRef(_))
23 }
24 }
25 }
1126}
View Options

src/main/scala/mesosphere/marathon/raml/NetworkConversion.scala

11package mesosphere.marathon
22package raml
33
44import mesosphere.marathon.core.pod
55import mesosphere.marathon.stream.Implicits._
66import mesosphere.mesos.protos.Implicits._
77import org.apache.mesos.Protos.ContainerInfo.DockerInfo.{ Network => DockerNetworkMode }
88
99trait NetworkConversion {
1010
11 import NetworkConversionMessages._
12
1113 implicit val networkRamlReader: Reads[Network, pod.Network] =
1214 Reads { raml =>
1315 raml.mode match {
1416 case NetworkMode.Host => pod.HostNetwork
1517 case NetworkMode.ContainerBridge => pod.BridgeNetwork(raml.labels)
1618 case NetworkMode.Container => pod.ContainerNetwork(
17 // TODO(PODS): shouldn't this be caught by validation?
18 raml.name.getOrElse(throw new IllegalArgumentException("container network must specify a name")),
19 // we expect validation to catch this problem first. but it's possible that migration
20 // also runs into this problem so we handle it explicitly.
21 raml.name.getOrElse(throw SerializationFailedException(ContainerNetworkRequiresName)),
1922 raml.labels
2023 )
2124 }
2225 }
2326
2427 implicit val networkRamlWriter: Writes[pod.Network, Network] = Writes {
2528 case cnet: pod.ContainerNetwork =>
2629 Network(
▲ Show 20 Lines • Show All 91 Lines • ▼ Show 20 Line(s)
118121 name = if (net.hasName) Option(net.getName) else Network.DefaultName,
119122 mode = mode,
120123 labels = if (net.getLabelsCount > 0) net.getLabelsList.to[Seq].fromProto else Network.DefaultLabels
121124 )
122125 }
123126}
124127
125128object NetworkConversion extends NetworkConversion
129
130object NetworkConversionMessages {
131 val ContainerNetworkRequiresName = "container network must specify a name"
132}
View Options

src/main/scala/mesosphere/marathon/storage/migration/MigrationTo1_5.scala

11package mesosphere.marathon
22package storage.migration
33
44import java.time.OffsetDateTime
55
66import akka.Done
77import akka.stream.Materializer
8import akka.stream.scaladsl.{ Sink, Source }
8import akka.stream.scaladsl.{ Flow, Keep, Sink, Source }
99import com.typesafe.scalalogging.StrictLogging
1010import mesosphere.marathon.Protos._
1111import mesosphere.marathon.api.v2.{ AppNormalization, AppsResource }
1212import mesosphere.marathon.raml.Raml
13import mesosphere.marathon.state.{ AppDefinition, PathId }
13import mesosphere.marathon.state.{ AppDefinition, PathId, RootGroup }
1414import mesosphere.marathon.storage.repository.GroupRepository
1515
1616import scala.async.Async.{ async, await }
1717import scala.concurrent.{ ExecutionContext, Future }
1818
1919case class MigrationTo1_5(
2020 migration: Migration)(implicit
2121 executionContext: ExecutionContext,
2222 materializer: Materializer) extends StrictLogging {
2323
2424 import MigrationTo1_5._
2525
2626 @SuppressWarnings(Array("all")) // async/await
2727 def migrate(): Future[Done] = async {
28 implicit val appNormalizer: Normalization[raml.App] = AppsResource.appNormalization(AppsResource.NormalizationConfig(
29 migration.availableFeatures,
30 new AppNormalization.Config {
31 override def defaultNetworkName: Option[String] =
32 sys.env.get(DefaultNetworkNameForMigratedApps).orElse(migration.defaultNetworkName).orElse(
33 throw SerializationFailedException(
34 "failed to migrate service because no default-network-name has been configured and" +
35 s" environment variable $DefaultNetworkNameForMigratedApps is not set")
36 )
37 }
38 ))
28 implicit val env = Environment(sys.env)
29 implicit val appNormalization = appNormalizer(migration.availableFeatures, migration.defaultNetworkName)
3930 val summary = await(migrateGroups(migration.serviceDefinitionRepo, migration.groupRepository))
4031 logger.info(s"Migrated $summary to 1.5")
4132 Done
4233 }
34}
4335
36private[migration] object MigrationTo1_5 {
37
38 val DefaultNetworkNameForMigratedApps = "MIGRATION_1_5_0_MARATHON_DEFAULT_NETWORK_NAME"
39 val MigrationFailedMissingNetworkEnvVar =
40 "failed to migrate service because no default-network-name has been configured and" +
41 s" environment variable $DefaultNetworkNameForMigratedApps is not set"
42
43 case class MigratedRoot(root: RootGroup, apps: Seq[AppDefinition]) {
44 def store(groupRepository: GroupRepository)(implicit ec: ExecutionContext): Future[MigratedRoot] =
45 groupRepository.storeRoot(root, apps, Nil, Nil, Nil).map(_ => MigratedRoot.this)
46 }
47
48 case class Environment(vars: Map[String, String])
49
50 def appNormalizer(enabledFeatures: Set[String], networkName: Option[String])(implicit env: Environment): Normalization[raml.App] =
jasongilanfarrUnsubmitted
Not Done

Should we validate too?

jdefAuthorUnsubmitted
Not Done

validation is baked into the AppsResource normalizer; no additional steps needed here

51 // lazily evaluate the special environment variable and configured network name: we might never need them, and in
52 // that case we don't want to abort migration (because there's no reason to).
53 AppsResource.appNormalization(AppsResource.NormalizationConfig(
54 enabledFeatures, new AppNormalization.Config {
55 override def defaultNetworkName: Option[String] =
56 env.vars.get(DefaultNetworkNameForMigratedApps).orElse(networkName).orElse(throw SerializationFailedException(
57 MigrationFailedMissingNetworkEnvVar))
58 }))
59
4460 /**
4561 * for each root version (+ current) load all apps from the service-definition-repository, migrate them,
4662 * then save changes for each root (and all of its apps) via the group-repository API.
4763 */
48 @SuppressWarnings(Array("all")) // async/await
49 def migrateGroups(serviceRepository: ServiceDefinitionRepository, groupRepository: GroupRepository)(implicit appNormalizer: Normalization[raml.App]): Future[(String, Int)] =
50 async {
51 val result: Future[(String, Int)] = groupRepository.rootVersions().mapAsync(Int.MaxValue) { version =>
64 def migrateGroups(
65 serviceRepository: ServiceDefinitionRepository,
66 groupRepository: GroupRepository)(implicit
67 appNormalizer: Normalization[raml.App],
68 ec: ExecutionContext,
69 mat: Materializer): Future[(String, Int)] =
70
71 groupRepository.rootVersions()
72 .via(loadRootsFlow(groupRepository))
73 .mapAsync[MigratedRoot](1)(migrateRoot(_, serviceRepository, groupRepository)) // store roots one at a time
74 .runWith(summarizeMigratedRootsSink)
75 .map { "root + app versions" -> _ }
76
77 /**
78 * load roots from the group repository, the flow always ends with the current root
79 */
80 def loadRootsFlow(groupRepository: GroupRepository) = Flow[OffsetDateTime].mapAsync(Int.MaxValue) { version =>
5281 groupRepository.rootVersion(version)
5382 }.collect {
5483 case Some(root) => root
5584 }.concat {
85 // ensure that the current root is written last. otherwise, after migration completes the resulting"current" root
aquamatthiasUnsubmitted
Not Done

Please add a short explanation, why current root must be last.

86 // may actually point to an older version (has to do with how the group repository stores roots and automatically
87 // adjusts the value of "current")
5688 Source.fromFuture(groupRepository.root())
57 }.mapAsync(1) { root =>
58 // store roots one at a time with current root last
89 }
90
91 /**
92 * migrate a root group by migrating all transitive apps and then storing the updated root group
93 */
94 def migrateRoot(
95 root: RootGroup,
96 serviceRepository: ServiceDefinitionRepository,
97 groupRepository: GroupRepository)(implicit
98 appNormalizer: Normalization[raml.App],
99 ec: ExecutionContext,
100 mat: Materializer): Future[MigratedRoot] = {
101
59102 val appIds: Seq[(PathId, OffsetDateTime)] = root.transitiveApps.map { app =>
60103 app.id -> app.version.toOffsetDateTime
61104 }(collection.breakOut)
62 serviceRepository.getVersions(appIds).map(migrateApp).runWith(Sink.seq).map { apps =>
63 groupRepository.storeRoot(root, apps, Nil, Nil, Nil).map(_ => apps.size)
105
106 serviceRepository.getVersions(appIds).via(migrateServiceFlow).runWith(Sink.seq)
107 .flatMap(MigratedRoot(root, _).store(groupRepository))
64108 }
65 }.flatMapConcat(f => Source.fromFuture(f)).runFold(0) {
66 case (acc, apps) => acc + apps + 1
67 }.map("root + app versions" -> _)
68 await(result)
69 }
70109
71 def migrateApp(service: ServiceDefinition)(implicit appNormalizer: Normalization[raml.App]): AppDefinition = {
110 /**
111 * migrate service definitions, first by converting from protobuf to RAML and then converting to the model API
112 */
113 def migrateServiceFlow(implicit appNormalizer: Normalization[raml.App]) = Flow[ServiceDefinition].map { service =>
72114 import Normalization._
73115 val rawRaml = Raml.toRaml(service)
74116 val normalizedApp = rawRaml.normalize
75117 val appDef = normalizedApp.fromRaml
76118 // fixup version since it's intentionally lost in the conversion from App to AppDefinition
77119 appDef.copy(versionInfo = AppDefinition.versionInfoFrom(service))
78120 }
79}
80121
81object MigrationTo1_5 {
82 private[migration] val DefaultNetworkNameForMigratedApps = "MIGRATION_1_5_0_MARATHON_DEFAULT_NETWORK_NAME"
122 def summarizeMigratedRootsSink = Flow[MigratedRoot].toMat(Sink.fold(0) {
123 case (acc, migratedRoot) =>
124 acc + migratedRoot.apps.size + 1 // number of apps migrated + 1 for the root
125 })(Keep.right)
126
83127}
View Options

src/main/scala/mesosphere/mesos/protos/Range.scala

11package mesosphere.mesos.protos
22
33import scala.collection.immutable.NumericRange
44
55case class Range(
66 begin: Long,
77 end: Long) {
88
99 def asScala(): NumericRange[Long] = {
1010 begin to end
1111 }
1212}
13
14object Range {
15 def apply(port: Long): Range = Range(port, port)
16}
View Options

src/main/scala/mesosphere/mesos/protos/RangesResource.scala

11package mesosphere.mesos.protos
22
3import scala.collection.immutable.Seq
4
35case class RangesResource(
46 name: String,
57 ranges: Seq[Range],
68 role: String = "*") extends Resource
9
10object RangesResource {
11 def ports(ranges: Seq[Range], role: String = "*"): RangesResource =
12 RangesResource(Resource.PORTS, ranges, role)
13}
View Options

src/main/scala/mesosphere/mesos/protos/ScalarResource.scala

11package mesosphere.mesos.protos
22
33case class ScalarResource(
44 name: String,
55 value: Double,
66 role: String = "*") extends Resource
7
8object ScalarResource {
9 def cpus(value: Double, role: String = "*"): ScalarResource = ScalarResource(Resource.CPUS, value, role)
10 def memory(value: Double, role: String = "*"): ScalarResource = ScalarResource(Resource.MEM, value, role)
11 def disk(value: Double, role: String = "*"): ScalarResource = ScalarResource(Resource.DISK, value, role)
12 def gpus(value: Double, role: String = "*"): ScalarResource = ScalarResource(Resource.GPUS, value, role)
13}
View Options

src/test/scala/mesosphere/marathon/api/v2/AppsResourceTest.scala

11package mesosphere.marathon
22package api.v2
33
44import java.util
55import javax.ws.rs.core.Response
66
77import akka.Done
88import mesosphere.AkkaUnitTest
99import mesosphere.marathon.api._
1010import mesosphere.marathon.core.appinfo.AppInfo.Embed
1111import mesosphere.marathon.core.appinfo._
1212import mesosphere.marathon.core.base.ConstantClock
1313import mesosphere.marathon.core.deployment.DeploymentPlan
1414import mesosphere.marathon.core.group.GroupManager
1515import mesosphere.marathon.core.plugin.PluginManager
1616import mesosphere.marathon.core.pod.ContainerNetwork
17import mesosphere.marathon.raml.{ App, AppUpdate, ContainerPortMapping, DockerContainer, DockerNetwork, EngineType, EnvVarValueOrSecret, IpAddress, IpDiscovery, IpDiscoveryPort, Network, NetworkMode, Raml, SecretDef }
17import mesosphere.marathon.raml.{ App, AppUpdate, ContainerPortMapping, DockerContainer, DockerNetwork, EngineType, EnvVarValueOrSecret, IpAddress, IpDiscovery, IpDiscoveryPort, Network, NetworkConversionMessages, NetworkMode, Raml, SecretDef }
1818import mesosphere.marathon.state.PathId._
1919import mesosphere.marathon.state._
2020import mesosphere.marathon.storage.repository.GroupRepository
2121import mesosphere.marathon.test.GroupCreation
2222import org.mockito.Matchers
2323import play.api.libs.json._
2424
2525import scala.collection.immutable
▲ Show 20 Lines • Show All 258 Lines • ▼ Show 20 Line(s)
284284 networks = Seq(Network(mode = NetworkMode.Container))
285285 )
286286
287287 When("The application is updated")
288288 val updatedJson = Json.toJson(updatedApp).as[JsObject]
289289 val updatedBody = Json.stringify(updatedJson).getBytes("UTF-8")
290290
291291 Then("the update should fail")
292 val caught = intercept[IllegalArgumentException] {
292 val caught = intercept[SerializationFailedException] {
293293 appsResource.replace(updatedApp.id, updatedBody, force = false, partialUpdate = false, auth.request)
294294 }
295 caught.getMessage() should be("container network must specify a name")
295 caught.getMessage() should be(NetworkConversionMessages.ContainerNetworkRequiresName)
296296 }
297297
298298 "Create a new app without IP/CT when default virtual network is bar" in new Fixture(configArgs = Seq("--default_network_name", "bar")) {
299299 Given("An app and group")
300300
301301 val app = App(
302302 id = "/app",
303303 cmd = Some("cmd")
▲ Show 20 Lines • Show All 1047 Lines • Show Last 20 Lines
View Options

src/test/scala/mesosphere/marathon/api/v2/json/AppDefinitionFormatsTest.scala

Show First 20 Lines • Show All 178 Lines • ▼ Show 20 Line(s)
179179 "FromJSON should not fail when 'cpus' is greater than 0" in {
180180 val json = Json.parse(""" { "id": "test", "cmd": "foo", "cpus": 0.0001 }""")
181181 noException should be thrownBy {
182182 normalizeAndConvert(json.as[raml.App])
183183 }
184184 }
185185
186186 "FromJSON should fail when 'env' contains invalid keys" in {
187 val json = Json.parse(""" { "id": "test", "cmd": "foo", "env": { "%^!": "foo" } }""")
187 val json = Json.parse(""" { "id": "test", "cmd": "foo", "env": { "": "foo" } }""")
188188 a[ValidationFailedException] shouldBe thrownBy { normalizeAndConvert(json.as[raml.App]) }
189189 }
190190
191191 """ToJSON should correctly handle missing acceptedResourceRoles""" in {
192192 val appDefinition = AppDefinition(id = PathId("test"), acceptedResourceRoles = Set.empty)
193193 val json = Json.toJson(appDefinition)
194194 (json \ "acceptedResourceRoles").asOpt[Set[String]] should be(None)
195195 }
▲ Show 20 Lines • Show All 481 Lines • Show Last 20 Lines
View Options

src/test/scala/mesosphere/marathon/api/v2/validation/EnvVarValidationTest.scala

Show All 9 Lines
1010class EnvVarValidationTest extends UnitTest with ResultMatchers with ValidationTestLike {
1111
1212 import Normalization._
1313 import EnvVarValidationMessages._
1414
1515 "EnvVarValidation" when {
1616 "an environment is validated" should {
1717
18 def compliantEnv(title: String, m: Map[String, EnvVarValueOrSecret]): Unit = {
19 s"$title is compliant with validation rules" in new WithoutSecrets {
18 def compliantEnv(title: String, m: Map[String, EnvVarValueOrSecret], strictNameValidation: Boolean = true): Unit = {
19 s"$title is compliant with validation rules" in new WithoutSecrets(strictNameValidation) {
2020 validate(m) should be(aSuccess)
2121 }
2222 }
2323
2424 behave like compliantEnv("empty env", Map.empty)
2525 behave like compliantEnv("singleton env", Environment("foo" -> "bar"))
2626 behave like compliantEnv("mixed case env", Environment("foo" -> "bar", "FOO" -> "BAR"))
2727 behave like compliantEnv("underscore env", Environment("_" -> "x"))
2828 behave like compliantEnv("alpha numerical env", Environment("a_1_" -> "x"))
29 behave like compliantEnv("whitespace env", Environment(" " -> "x"), strictNameValidation = false)
30 behave like compliantEnv("hyphen env", Environment("-" -> "x"), strictNameValidation = false)
31 behave like compliantEnv("numerical env", Environment("9" -> "x"), strictNameValidation = false)
2932
3033 "fail with a numerical env variable name" in new WithoutSecrets {
3134 validate(Wrapper(Environment("9" -> "x"))).normalize should failWith("/env(9)" -> MustContainOnlyAlphanumeric)
3235 }
3336
34 "fail with empty variable name" in new WithoutSecrets {
35 validate(Wrapper(Environment("" -> "x"))).normalize should failWith(
36 "/env()" -> "must not be empty",
37 "/env()" -> MustContainOnlyAlphanumeric
38 )
37 def failsWhenExpected(subtitle: String, strictNameValidation: Boolean): Unit = {
38 s"fail with empty variable name $subtitle" in new WithoutSecrets(strictNameValidation) {
39 val alwaysExpected: Seq[ViolationMatcher] = Seq("/env()" -> "must not be empty")
40 val expectedNow =
41 if (strictNameValidation) alwaysExpected ++ (Seq[ViolationMatcher]("/env()" -> MustContainOnlyAlphanumeric))
42 else alwaysExpected
43
44 validate(Wrapper(Environment("" -> "x"))).normalize should failWith(expectedNow: _*)
3945 }
4046
41 "fail with too long variable name" in new WithoutSecrets {
47 s"fail with too long variable name $subtitle" in new WithoutSecrets(strictNameValidation) {
4248 val name = ("x" * 255)
4349 validate(Wrapper(Environment(name -> "x"))).normalize should failWith(
4450 s"/env($name)" -> VariableNameTooLong
4551 )
4652 }
4753 }
54
55 behave like failsWhenExpected("(strict)", true)
56 behave like failsWhenExpected("(non-strict)", false)
4857 }
58 }
4959
50 class WithoutSecrets {
60 class WithoutSecrets(strictNameValidation: Boolean = true) {
5161 implicit lazy val envVarValidation: Validator[Map[String, EnvVarValueOrSecret]] =
52 EnvVarValidation.envValidator(Map.empty, Set.empty)
62 EnvVarValidation.envValidator(strictNameValidation, Map.empty, Set.empty)
5363
5464 case class Wrapper(env: Map[String, EnvVarValueOrSecret])
5565
5666 object Wrapper {
5767 implicit val wrapperValidation: Validator[Wrapper] = validator { wrapper =>
5868 wrapper.env is envVarValidation // invoked here the same way that it is for apps and pods
5969 }
6070 }
6171 }
6272}
View Options

src/test/scala/mesosphere/marathon/raml/AppConversionTest.scala

Show First 20 Lines • Show All 134 Lines • ▼ Show 20 Line(s)
135135 residency = Option(AppResidency(
136136 relaunchEscalationTimeoutSeconds = 33,
137137 taskLostBehavior = TaskLostBehavior.RelaunchAfterTimeout
138138 )),
139139 versionInfo = Option(VersionInfo(
140140 lastScalingAt = Timestamp.zero.toOffsetDateTime,
141141 lastConfigChangeAt = Timestamp.zero.toOffsetDateTime
142142 )),
143 portDefinitions = Option(Seq.empty)
143 portDefinitions = None
144144 )
145145 legacy.toRaml[App] should be(expectedRaml)
146146 }
147147 }
148148}
View Options

src/test/scala/mesosphere/marathon/raml/ContainerConversionTest.scala

Show First 20 Lines • Show All 187 Lines • ▼ Show 20 Line(s)
188188 containerPort = 1,
189189 hostPort = Option(2),
190190 labels = Map("foo" -> "bar"),
191191 name = Option("http"),
192192 servicePort = 3
193193 ))
194194 )
195195 )),
196 portMappings = Option(Seq.empty)
196 portMappings = None
197197 )
198198 legacyProto.toRaml[Container] should be(expectedRaml)
199199 }
200200 }
201201 "a docker-docker container" should {
202202 val container = state.Container.Docker(Seq(coreHostVolume), "test", Seq(corePortMapping))
203203 val raml = container.toRaml[Container]
204204
▲ Show 20 Lines • Show All 53 Lines • Show Last 20 Lines
View Options

src/test/scala/mesosphere/marathon/raml/EnvVarConversionTest.scala

  • This file was added.
1package mesosphere.marathon
2package raml
3
4import mesosphere.UnitTest
5import mesosphere.marathon.stream.Implicits._
6import mesosphere.mesos.protos.Implicits._
7import org.apache.mesos.{ Protos => Mesos }
8
9class EnvVarConversionTest extends UnitTest {
10
11 import Environment.Implicits._
12
13 "EnvVarConversion" when {
14 "converting from proto to RAML" should {
15 def convertEnv(subtitle: String, sdf: Fixture => Protos.ServiceDefinition, expected: Map[String, EnvVarValueOrSecret]): Unit =
16 s"succeed when $subtitle" in new Fixture {
17 val sd: Protos.ServiceDefinition = sdf(this)
18 val converted = sd.whenOrElse(
19 _.hasCmd,
20 s => (s.getCmd.getEnvironment.getVariablesList.to[Seq], s.getEnvVarReferencesList.to[Seq]).toRaml,
21 App.DefaultEnv)
22 converted should be(expected)
23 }
24
25 behave like convertEnv("no env present", _.emptyService, Map.empty)
26
27 val simpleVars = Map("a" -> "b", "q" -> "z")
28 behave like convertEnv("simple vars present", _.withVars(vars = simpleVars), Environment(simpleVars))
29
30 val secretVars = Map("var1" -> "secret1", "var2" -> "secret2")
31 behave like convertEnv("only secrets present", _.withSecrets(secretRefs = secretVars), Environment().withSecrets(secretVars))
32
33 behave like convertEnv(
34 "simple and secret refs present",
35 f => f.withVars(f.withSecrets(secretRefs = secretVars), vars = simpleVars),
36 Environment(simpleVars).withSecrets(secretVars)
37 )
38 }
39 }
40
41 class Fixture {
42 val emptyService: Protos.ServiceDefinition = Protos.ServiceDefinition.newBuilder()
43 .setCmd(Mesos.CommandInfo.newBuilder())
44 .setExecutor("//cmd")
45 .setId("/foo")
46 .setInstances(1)
47 .build
48
49 def withVars(sd: Protos.ServiceDefinition = emptyService, vars: Map[String, String]): Protos.ServiceDefinition = {
50 val env = if (sd.getCmd.hasEnvironment) sd.getCmd.getEnvironment.toBuilder else Mesos.Environment.newBuilder()
51 vars.foreach {
52 case (k, v) =>
53 env.addVariables(Mesos.Environment.Variable.newBuilder().setName(k).setValue(v))
54 }
55 val cmd = sd.getCmd.toBuilder
56 cmd.setEnvironment(env)
57 sd.toBuilder.setCmd(cmd).build()
58 }
59
60 def withSecrets(sd: Protos.ServiceDefinition = emptyService, secretRefs: Map[String, String]): Protos.ServiceDefinition = {
61 val builder = sd.toBuilder
62 secretRefs.foreach {
63 case (envVarName, secretName) =>
64 builder.addEnvVarReferencesBuilder()
65 .setType(Protos.EnvVarReference.Type.SECRET)
66 .setName(envVarName)
67 .setSecretRef(Protos.EnvVarSecretRef.newBuilder().setSecretId(secretName))
68 }
69 builder.build
70 }
71 }
72
73}
View Options

src/test/scala/mesosphere/marathon/storage/migration/MigrationTo1_5Test.scala

  • This file was added.
1package mesosphere.marathon
2package storage.migration
3
4import java.time.OffsetDateTime
5
6import akka.Done
7import akka.stream.scaladsl.{ Sink, Source }
8import mesosphere.AkkaUnitTest
9import mesosphere.marathon.core.pod.{ BridgeNetwork, ContainerNetwork }
10import mesosphere.marathon.raml.Resources
11import mesosphere.marathon.state.VersionInfo.{ FullVersionInfo, OnlyVersion }
12import mesosphere.marathon.state._
13import mesosphere.marathon.storage.repository.GroupRepository
14import mesosphere.marathon.stream.Implicits._
15import mesosphere.marathon.test.GroupCreation
16import mesosphere.mesos.protos.Implicits._
17import mesosphere.mesos.protos.{ Range, RangesResource, ScalarResource }
18import org.apache.mesos.{ Protos => Mesos }
19import org.scalatest.RecoverMethods
20
21import scala.concurrent.Future
22
23class MigrationTo1_5Test extends AkkaUnitTest with RecoverMethods with GroupCreation {
24
25 import MigrationTo1_5._
26 import Protos.ExtendedContainerInfo.DockerInfo
27
28 private def migrateSingleApp(sd: Protos.ServiceDefinition)(implicit n: Normalization[raml.App]): AppDefinition =
29 migrateSingleAppF(sd).futureValue
30
31 private def migrateSingleAppF(sd: Protos.ServiceDefinition)(implicit n: Normalization[raml.App]): Future[AppDefinition] =
32 Source.single(sd).via(migrateServiceFlow).runWith(Sink.head)
33
34 "MigrationTo1_5" when {
35
36 "migrating a single app" should {
37
38 "basic command" in new Fixture {
39 migrateSingleApp(basicCommandService) should be(basicCommandApp)
40 }
41
42 "scale+config versions" in new Fixture {
43 val sd = basicCommandService.toBuilder
44 .setLastScalingAt(2000L)
45 .setLastConfigChangeAt(3000L)
46 .build
47 val expected = basicCommandApp.copy(
48 versionInfo = FullVersionInfo(
49 version = basicCommandApp.version,
50 lastScalingAt = Timestamp(2000L),
51 lastConfigChangeAt = Timestamp(3000L)
52 )
53 )
54 migrateSingleApp(sd) should be(expected)
55 }
56
57 "mesos container, host networking" in new Fixture {
58 val sd = withContainer { containerInfo =>
59 containerInfo.setType(Mesos.ContainerInfo.Type.MESOS)
60 containerInfo.addVolumes(Protos.Volume.newBuilder()
61 .setHostPath("/host")
62 .setContainerPath("containerPath")
63 .setMode(Mesos.Volume.Mode.RW)
64 .build
65 )
66 }
67 val expected = basicCommandApp.copy(
68 container = Some(Container.Mesos(Seq(DockerVolume("containerPath", "/host", Mesos.Volume.Mode.RW))))
69 )
70 migrateSingleApp(sd) should be(expected)
71 }
72
73 "mesos container, host networking, w/o ports" in new Fixture {
74 val sd = withContainer(_.setType(Mesos.ContainerInfo.Type.MESOS)).toBuilder.clearPortDefinitions().build
75 val expected = basicCommandApp.copy(container = Some(Container.Mesos()), portDefinitions = Nil)
76 migrateSingleApp(sd) should be(expected)
77 }
78
79 def failsWithNetworkNameRequired(title: String)(f: Fixture => Protos.ServiceDefinition) = {
80 title in new Fixture {
81 val sd: Protos.ServiceDefinition = f(this)
82
83 recoverToExceptionIf[SerializationFailedException] {
84 migrateSingleAppF(sd)
85 }.map { ex =>
86 ex.getMessage should be(MigrationFailedMissingNetworkEnvVar)
87 }.futureValue
88 }
89 }
90
91 behave like failsWithNetworkNameRequired("mesos container, IP/CT networking (unnamed), w/o ports"){ fixture =>
92 fixture.withDeprecatedIpAddress(fixture.withContainer(_.setType(Mesos.ContainerInfo.Type.MESOS)), ports = Nil)
93 }
94
95 behave like failsWithNetworkNameRequired("mesos container, IP/CT networking (unnamed)"){ fixture =>
96 fixture.withDeprecatedIpAddress(fixture.withContainer(_.setType(Mesos.ContainerInfo.Type.MESOS)))
97 }
98
99 "mesos container, IP/CT networking (named), w/o ports" in new Fixture {
100 val sd = withDeprecatedIpAddress(
101 withContainer(_.setType(Mesos.ContainerInfo.Type.MESOS)),
102 Option("someNetworkName"), ports = Nil)
103 val expected = basicCommandApp.copy(
104 container = Some(Container.Mesos()),
105 networks = Seq(ContainerNetwork(name = "someNetworkName")),
106 portDefinitions = Nil
107 )
108 migrateSingleApp(sd) should be(expected)
109 }
110
111 "mesos container, IP/CT networking (named)" in new Fixture {
112 val sd = withDeprecatedIpAddress(
113 withContainer(_.setType(Mesos.ContainerInfo.Type.MESOS)),
114 Option("someNetworkName"))
115 val expected = basicCommandApp.copy(
116 container = Some(Container.Mesos(
117 portMappings = Seq(Container.PortMapping(containerPort = 80, name = Some("http")))
118 )),
119 networks = Seq(ContainerNetwork(name = "someNetworkName")),
120 portDefinitions = Nil
121 )
122 migrateSingleApp(sd) should be(expected)
123 }
124
125 "docker container, host networking" in new Fixture {
126 val sd = withContainer(
127 _.setType(Mesos.ContainerInfo.Type.DOCKER).setDocker(DockerInfo.newBuilder()
128 .setImage("image0").setOBSOLETENetwork(Mesos.ContainerInfo.DockerInfo.Network.HOST)))
129 val expected = basicCommandApp.copy(container = Some(Container.Docker(image = "image0")))
130 migrateSingleApp(sd) should be(expected)
131 }
132
133 "docker container, host networking, w/o ports" in new Fixture {
134 val sd = withContainer(
135 _.setType(Mesos.ContainerInfo.Type.DOCKER).setDocker(DockerInfo.newBuilder()
136 .setImage("image0").setOBSOLETENetwork(Mesos.ContainerInfo.DockerInfo.Network.HOST))
137 ).toBuilder.clearPortDefinitions().build
138
139 val expected = basicCommandApp.copy(container = Some(Container.Docker(image = "image0")), portDefinitions = Nil)
140 migrateSingleApp(sd) should be(expected)
141 }
142
143 "docker container, bridge networking" in new Fixture {
144 val sd = withContainer(
145 _.setType(Mesos.ContainerInfo.Type.DOCKER).setDocker(DockerInfo.newBuilder()
146 .setImage("image0")
147 .setOBSOLETENetwork(Mesos.ContainerInfo.DockerInfo.Network.BRIDGE)
148 .addOBSOLETEPortMappings(DockerInfo.ObsoleteDockerPortMapping.newBuilder()
149 .setContainerPort(8080)
150 .setHostPort(123)
151 .setServicePort(456)
152 .setName("rtp")
153 .setProtocol("udp")
154 )
155 )).toBuilder.clearPortDefinitions().build
156 val expected = basicCommandApp.copy(
157 container = Some(Container.Docker(image = "image0", portMappings = Seq(
158 Container.PortMapping(
159 containerPort = 8080,
160 hostPort = Option(123),
161 servicePort = 456,
162 name = Option("rtp"),
163 protocol = "udp"
164 )
165 ))),
166 networks = Seq(BridgeNetwork()),
167 portDefinitions = Nil
168 )
169 migrateSingleApp(sd) should be(expected)
170 }
171
172 "docker container, bridge networking, w/o ports" in new Fixture {
173 val sd = withContainer(
174 _.setType(Mesos.ContainerInfo.Type.DOCKER).setDocker(DockerInfo.newBuilder()
175 .setImage("image0")
176 .setOBSOLETENetwork(Mesos.ContainerInfo.DockerInfo.Network.BRIDGE)
177 )).toBuilder.clearPortDefinitions().build
178
179 val expected = basicCommandApp.copy(
180 container = Some(Container.Docker(image = "image0")),
181 networks = Seq(BridgeNetwork()),
182 portDefinitions = Nil
183 )
184 migrateSingleApp(sd) should be(expected)
185 }
186
187 "docker container, user networking (named)" in new Fixture {
188 val sd = withDeprecatedIpAddress(withContainer(
189 _.setType(Mesos.ContainerInfo.Type.DOCKER).setDocker(DockerInfo.newBuilder()
190 .setImage("image0")
191 .setOBSOLETENetwork(Mesos.ContainerInfo.DockerInfo.Network.USER)
192 .addOBSOLETEPortMappings(DockerInfo.ObsoleteDockerPortMapping.newBuilder()
193 .setContainerPort(8080)
194 .setHostPort(123)
195 .setServicePort(456)
196 .setName("rtp")
197 .setProtocol("udp")
198 )
199 )), networkName = Some("someNetworkName"), ports = Nil)
200
201 val expected = basicCommandApp.copy(
202 container = Some(Container.Docker(image = "image0", portMappings = Seq(
203 Container.PortMapping(
204 containerPort = 8080,
205 hostPort = Option(123),
206 servicePort = 456,
207 name = Option("rtp"),
208 protocol = "udp"
209 )
210 ))),
211 networks = Seq(ContainerNetwork("someNetworkName")),
212 portDefinitions = Nil
213 )
214 migrateSingleApp(sd) should be(expected)
215 }
216
217 "docker container, user networking (named), w/o ports" in new Fixture {
218 val sd = withDeprecatedIpAddress(withContainer(
219 _.setType(Mesos.ContainerInfo.Type.DOCKER).setDocker(DockerInfo.newBuilder()
220 .setImage("image0")
221 .setOBSOLETENetwork(Mesos.ContainerInfo.DockerInfo.Network.USER)
222 )), networkName = Some("someNetworkName"), ports = Nil)
223
224 val expected = basicCommandApp.copy(
225 container = Some(Container.Docker(image = "image0")),
226 networks = Seq(ContainerNetwork("someNetworkName")),
227 portDefinitions = Nil
228 )
229 migrateSingleApp(sd) should be(expected)
230 }
231
232 behave like failsWithNetworkNameRequired("docker container, user networking (unnamed)"){ fixture =>
233 fixture.withDeprecatedIpAddress(fixture.withContainer(
234 _.setType(Mesos.ContainerInfo.Type.DOCKER).setDocker(DockerInfo.newBuilder()
235 .setImage("image0")
236 .setOBSOLETENetwork(Mesos.ContainerInfo.DockerInfo.Network.USER)
237 )), ports = Nil)
238 }
239
240 def migrationWithFallbackNetwork(subtitle: String, f: Fixture): Unit = {
241 import f._
242
243 s"docker container, user networking (unnamed), w/o ports, $subtitle" in {
244 val sd = withDeprecatedIpAddress(withContainer(
245 _.setType(Mesos.ContainerInfo.Type.DOCKER).setDocker(DockerInfo.newBuilder()
246 .setImage("image0")
247 .setOBSOLETENetwork(Mesos.ContainerInfo.DockerInfo.Network.USER)
248 )), ports = Nil)
249
250 val expected = basicCommandApp.copy(
251 container = Some(Container.Docker(image = "image0")),
252 networks = Seq(ContainerNetwork("defaultNetwork")),
253 portDefinitions = Nil
254 )
255 migrateSingleApp(sd) should be(expected)
256 }
257
258 s"mesos container, IP/CT networking (unnamed), w/o ports, $subtitle" in {
259 val sd = withDeprecatedIpAddress(withContainer(_.setType(Mesos.ContainerInfo.Type.MESOS)), ports = Nil)
260 val expected = basicCommandApp.copy(
261 container = Some(Container.Mesos()),
262 networks = Seq(ContainerNetwork(name = "defaultNetwork")),
263 portDefinitions = Nil
264 )
265 migrateSingleApp(sd) should be(expected)
266 }
267 }
268
269 behave like migrationWithFallbackNetwork("w/ default network", new Fixture(
270 defaultNetworkName = Some("defaultNetwork")))
271
272 behave like migrationWithFallbackNetwork("w/ migration envvar override", new Fixture(
273 env = Map(DefaultNetworkNameForMigratedApps -> "defaultNetwork")))
274
275 behave like migrationWithFallbackNetwork("w/ migration envvar that overrides command line default", new Fixture(
276 defaultNetworkName = Some("foobar"), env = Map(DefaultNetworkNameForMigratedApps -> "defaultNetwork")))
277 }
278
279 "migrating a root group" should {
280 "be a noop when the root group is empty" in new Fixture {
281 val emptyRoot = createRootGroup(
282 apps = Map.empty,
283 pods = Map.empty,
284 groups = Set.empty,
285 dependencies = Set.empty,
286 version = Timestamp.zero
287 )
288
289 groupRepository.rootVersions() returns Source.empty[OffsetDateTime]
290 groupRepository.root() returns Future.successful(emptyRoot)
291 serviceDefinitionRepository.getVersions(any) returns Source.empty[Protos.ServiceDefinition]
292 groupRepository.storeRoot(any, any, any, any, any) returns Future.successful(Done)
293
294 val futureSummary = migrateGroups(serviceDefinitionRepository, groupRepository)
295 val (_, count) = futureSummary.futureValue
296 count should be(1)
297
298 verify(groupRepository).rootVersions()
299 verify(groupRepository).root()
300 verify(serviceDefinitionRepository).getVersions(Nil)
301 verify(groupRepository).storeRoot(emptyRoot, Nil, Nil, Nil, Nil)
302 noMoreInteractions(groupRepository, serviceDefinitionRepository)
303 }
304 "migrate all apps in the current root group" in new Fixture {
305 val singleAppRoot = createRootGroup(
306 apps = Map(basicCommandApp.id -> AppDefinition(id = basicCommandApp.id, versionInfo = basicCommandApp.versionInfo)),
307 pods = Map.empty,
308 groups = Set.empty,
309 dependencies = Set.empty,
310 version = Timestamp.zero
311 )
312
313 groupRepository.rootVersions() returns Source.empty[OffsetDateTime]
314 groupRepository.root() returns Future.successful(singleAppRoot)
315 serviceDefinitionRepository.getVersions(any) returns Source.single(basicCommandService)
316 groupRepository.storeRoot(any, any, any, any, any) returns Future.successful(Done)
317
318 val futureSummary = migrateGroups(serviceDefinitionRepository, groupRepository)
319 val (_, count) = futureSummary.futureValue
320 count should be(2) // root + 1 app
321
322 verify(groupRepository).rootVersions()
323 verify(groupRepository).root()
324 verify(serviceDefinitionRepository).getVersions(Seq(basicCommandApp.id -> serviceVersion.toOffsetDateTime))
325 verify(groupRepository).storeRoot(singleAppRoot, Seq(basicCommandApp), Nil, Nil, Nil)
326 noMoreInteractions(groupRepository, serviceDefinitionRepository)
327 }
328 "migrate root version, followed by current" in new Fixture {
329 val emptyRoot = createRootGroup(
330 apps = Map.empty,
331 pods = Map.empty,
332 groups = Set.empty,
333 dependencies = Set.empty,
334 version = Timestamp.zero
335 )
336 val singleAppRoot = createRootGroup(
337 apps = Map(basicCommandApp.id -> AppDefinition(id = basicCommandApp.id, versionInfo = basicCommandApp.versionInfo)),
338 pods = Map.empty,
339 groups = Set.empty,
340 dependencies = Set.empty,
341 version = Timestamp(1L)
342 )
343
344 groupRepository.rootVersions() returns Source.single(Timestamp.zero.toOffsetDateTime)
345 groupRepository.rootVersion(any) returns Future.successful(Option(emptyRoot))
346 groupRepository.root() returns Future.successful(singleAppRoot)
347
348 serviceDefinitionRepository.getVersions(Nil) returns Source.empty[Protos.ServiceDefinition]
349 val serviceVersions = Seq(basicCommandApp.id -> serviceVersion.toOffsetDateTime)
350 serviceDefinitionRepository.getVersions(serviceVersions) returns Source.single(basicCommandService)
351 groupRepository.storeRoot(any, any, any, any, any) returns Future.successful(Done)
352
353 val futureSummary = migrateGroups(serviceDefinitionRepository, groupRepository)
354 val (_, count) = futureSummary.futureValue
355 count should be(3) // versioned-root (empty) + current root + current app
356
357 verify(groupRepository).rootVersions()
358
359 verify(groupRepository).rootVersion(Timestamp.zero.toOffsetDateTime)
360 verify(serviceDefinitionRepository).getVersions(Nil)
361 verify(groupRepository).storeRoot(emptyRoot, Nil, Nil, Nil, Nil)
362
363 verify(groupRepository).root()
364 verify(serviceDefinitionRepository).getVersions(serviceVersions)
365 verify(groupRepository).storeRoot(singleAppRoot, Seq(basicCommandApp), Nil, Nil, Nil)
366
367 noMoreInteractions(groupRepository, serviceDefinitionRepository)
368 }
369 }
370 }
371
372 private class Fixture(val defaultNetworkName: Option[String] = None, val env: Map[String, String] = Map.empty) {
373 val serviceVersion = Timestamp(1000L)
374 val basicCommandService: Protos.ServiceDefinition = Protos.ServiceDefinition.newBuilder()
375 .setId("/foo")
376 .setCmd(Mesos.CommandInfo.newBuilder().setValue("sleep 60"))
377 .setInstances(1)
378 .setExecutor("//cmd")
379 .addAllResources(Seq(
380 ScalarResource.cpus(0.1),
381 ScalarResource.memory(64),
382 ScalarResource.disk(10),
383 ScalarResource.gpus(0),
384 RangesResource.ports(Seq(Range(80)))
385 ).map(resourceToProto))
386 .addPortDefinitions(Mesos.Port.newBuilder().setNumber(80))
387 .setVersion(serviceVersion.toString)
388 .build
389
390 val basicCommandApp: AppDefinition = AppDefinition(
391 id = PathId("/foo"),
392 cmd = Option("sleep 60"),
393 resources = Resources(cpus = 0.1, mem = 64, disk = 10),
394 executor = "//cmd",
395 portDefinitions = PortDefinitions(80),
396 versionInfo = OnlyVersion(serviceVersion)
397 )
398
399 def withContainer(f: Protos.ExtendedContainerInfo.Builder => Protos.ExtendedContainerInfo.Builder): Protos.ServiceDefinition = {
400 basicCommandService.toBuilder
401 .setContainer(f(Protos.ExtendedContainerInfo.newBuilder()).build)
402 .build
403 }
404
405 def withDeprecatedIpAddress(sd: Protos.ServiceDefinition, networkName: Option[String] = None, ports: Seq[(Int, String)] = Seq(80 -> "http")) = {
406 val ipAddress = Protos.ObsoleteIpAddress.newBuilder()
407 .setDiscoveryInfo(Protos.ObsoleteDiscoveryInfo.newBuilder()
408 .addAllPorts(ports.map { case (port, name) => Mesos.Port.newBuilder().setNumber(port).setName(name).build })
409 )
410 networkName.map(ipAddress.setNetworkName)
411 sd.toBuilder.setOBSOLETEIpAddress(ipAddress).clearPortDefinitions().build
412 }
413
414 lazy implicit val environment: Environment = Environment(env)
415 lazy implicit val appNormalization: Normalization[raml.App] = appNormalizer(Set.empty, defaultNetworkName)
416
417 val serviceDefinitionRepository = mock[ServiceDefinitionRepository]
418 val groupRepository = mock[GroupRepository]
419 }
420}
View Options

src/test/scala/mesosphere/mesos/protos/ImplicitsSpec.scala

11package mesosphere.mesos.protos
22
33import mesosphere.UnitTest
44import org.apache.mesos.Protos
55
6import scala.collection.immutable.Seq
aquamatthiasUnsubmitted
Not Done

Nit: please use double package notation. You should not need this import then.

jdefAuthorUnsubmitted
Not Done

double package only applies for mesosphere.marathon packages

7
68class ImplicitsSpec extends UnitTest {
79
810 import Implicits._
911
1012 "Implicits" should {
1113 "convert ExecutorID to proto and back" in {
1214 val caseClass = ExecutorID("klaus")
1315 assert(caseClass == ((caseClass: Protos.ExecutorID): ExecutorID))
▲ Show 20 Lines • Show All 78 Lines • Show Last 20 Lines