HomeMesosphereNo notifications. 4 unresolved issues.
  • Search
Differential D773

Support passing a Docker config.json for app / pod being launched
ClosedAll Users
Actions

Authored by ichernetsky on May 10 2017, 2:44 PM.

Details

Reviewers
jenkins
meichstedt
aquamatthias
zen-dog
kensipe
unterstein
jdef
jeschkies
Commits
rMARATHON9450c8e37911: More of testing and validating Docker Pull Config API
rMARATHONa30d8c678ead: Address @aquamatthias' comments
rMARATHON359bae165365: Document Docker image config option
rMARATHON1ee454c46811: Add support of Docker Image config field for pods too
rMARATHON4896abb92cc5: Introduce ImagePullConfig proto message
rMARATHONe43d34540e72: Make sure that config option is used only with both Mesos Containerizer and…
rMARATHON78bb703eeeba: Introduce ImagePullConfig proto message
rMARATHON34bf51bd508e: Support passing a Docker config for task / pod being launched
rMARATHONde2e9bbd060a: Address @jdef's comments
rMARATHONa3c4dab98dd4: Address @jdef's comments
rMARATHONf452d7149626: Address @jdef's comments and use the existing secret interface instead of…
rMARATHONd66c2c00541e: Support passing a Docker config for task / pod being launched
rMARATHONc6bbd6576e5a: Document Docker image config option
rMARATHON4ac015623b5d: Uncomment a plugin
rMARATHON41bf6e4c7e70: Fix a test
rMARATHON37f1068e05f8: Compile *.proto files with protoc v2.6.1
rMARATHON01acfd8593f3: Support container.docker.config.{text,secret}
rMARATHONcc9e43d9b5d8: SI test for a pod pulling an image from a private registry
rMARATHONf6b6cdf5ea8d: SI test for a pod pulling an image from a private registry
rMARATHON32756ea6909a: Fix a test
rMARATHON05ad39208c76: Tell people use protoc v2.6.1
rMARATHON825d33d43e88: Compile *.proto files with protoc v2.6.1
rMARATHON8a2472d3cbcb: Update integration tests to test fetching of private Docker images
rMARATHONe6b229eb08ec: Address comments
rMARATHONdb79c855b81f: Copy mesos.proto from Mesos' master branch (commit…
rMARATHON5d7dfd6613ec: Update integration tests to test fetching of private Docker images
rMARATHONe31af6c1d61d: Address comments for docs
rMARATHONb97782ca9309: Support passing a Docker config.json for app / pod being launched
rMARATHON58363801d813: Drop support for "docker.config.text" and use SecretDef for "docker.config.
rMARATHONea89b203b5b8: Tell people use protoc v2.6.1
rMARATHON1a0540fac9ab: Uncomment a plugin
rMARATHON4013d6404a35: Address comments for docs
rMARATHON39bec377ba9c: Drop support for "docker.config.text" and use SecretDef for "docker.config.
rMARATHONb0e91b9ab642: Make sure that config option is used only with both Mesos Containerizer and…
rMARATHONe5e3817775c5: Replace config with pullConfig in native-docker.md
rMARATHON7795b89a8ecb: Use Mesos 1.4.0-SNAPSHOT
rMARATHON297c502ed930: Support passing a Docker config.json for app / pod being launched
rMARATHONe994b756a437: Support passing a Docker config.json for app / pod being launched
rMARATHON2d1c9c32e46d: SI test for a pod pulling an image from a private registry
rMARATHONca27cb5a7127: Address comments for docs
rMARATHONb0ca2be54a47: More of testing and validating Docker Pull Config API
rMARATHONeb0fd4b85d09: Use Mesos 1.4.0-SNAPSHOT
rMARATHON9d0901eb3402: Support passing a Docker config.json for app / pod being launched
rMARATHON58908feb732d: Address @jdef's comments
rMARATHON5c6cc18e6a2c: Add support of Docker Image config field for pods too
rMARATHONf3e890b0d545: Address @jdef's comments
rMARATHONebe73d447582: Support container.docker.config.{text,secret}
rMARATHON0e5d61d96a8f: Use Mesos 1.3.0-rc3
rMARATHON487e006cbe26: Address @aquamatthias' comments
rMARATHON7d49bc18164b: Bring container.docker.credential back to life, but deprecate it
rMARATHONb37a5c26b3b0: Address @jdef's comments and use the existing secret interface instead of…
rMARATHON084b07b1df64: Bring container.docker.credential back to life, but deprecate it
rMARATHON4b0523692730: Copy mesos.proto from Mesos' master branch (commit…
JIRA Issues
JIRA MARATHON-7090 UCR Authentication per task / docker registry
Test Plan

Run tests, launch an app with "container.docker.config" set

Diff Detail

Repository
rMARATHON marathon
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
There are a very large number of changes, so older changes are hidden. Show Older Changes
ichernetsky added a comment.May 24 2017, 9:21 AM
  • Replace config with pullConfig in native-docker.md
  • Address @jdef's comments
jenkins accepted this revision.May 24 2017, 9:38 AM

✔ Build of 3328 completed jenkins-public-marathon-phabricator-87.

You can create a DC/OS with your patched Marathon by creating a new pull
request with the following changes in buildinfo.json:

"url": "https://downloads.mesosphere.io/marathon/snapshots/marathon-1.5.0-SNAPSHOT-538-g44f7e23.tgz",
"sha1"" "67f4f9849ae3b340c7aa357d8c2d9363904bc987"
ichernetsky marked an inline comment as done.May 24 2017, 7:35 PM
ichernetsky updated this revision to Diff 3340.May 24 2017, 8:41 PM
  • Introduce ImagePullConfig proto message
jenkins accepted this revision.May 24 2017, 9:02 PM

✔ Build of 3340 completed jenkins-public-marathon-phabricator-101.

You can create a DC/OS with your patched Marathon by creating a new pull
request with the following changes in buildinfo.json:

"url": "https://downloads.mesosphere.io/marathon/snapshots/marathon-1.5.0-SNAPSHOT-538-g44f7e23.tgz",
"sha1"" "d00b47ad3634bac6475ff56b2f62354f0cd95100"
ichernetsky updated this revision to Diff 3341.May 25 2017, 5:53 AM
  • More of testing and validating Docker Pull Config API
jenkins accepted this revision.May 25 2017, 6:07 AM

✔ Build of 3341 completed jenkins-public-marathon-phabricator-102.

You can create a DC/OS with your patched Marathon by creating a new pull
request with the following changes in buildinfo.json:

"url": "https://downloads.mesosphere.io/marathon/snapshots/marathon-1.5.0-SNAPSHOT-538-g44f7e23.tgz",
"sha1"" "cd497bd39061b5d0667d76ff998bcc5f1d073502"
jenkins requested changes to this revision.May 25 2017, 5:29 PM
This revision now requires changes to proceed.May 25 2017, 5:29 PM
jenkins added a comment.May 25 2017, 5:29 PM

✗ Build of 3342 failed jenkins-public-marathon-phabricator-103.
Error message:

Stage Compile and Test failed.

jdef added inline comments.May 25 2017, 10:12 PM
docs/docs/rest-api/public/api/v2/types/appContainer.raml
15

???

119

i'm looking for "Deprecated in favor of pullConfig"

project/Dependencies.scala
86

I've heard that there's a 1.3.0-rc1 that contains what we need?

src/main/scala/mesosphere/marathon/api/serialization/ContainerSerializer.scala
341

nit: should not need new here

src/main/scala/mesosphere/marathon/api/v2/validation/AppValidation.scala
55

i don't see a validation test case for this. is it there and missed it? if not, please add one

src/main/scala/mesosphere/marathon/api/v2/validation/PodsValidation.scala
134

this case _ is superfluous since ImageType is a sealed trait

158

wow, we really left abc in there, eh?

src/main/scala/mesosphere/marathon/raml/ContainerConversion.scala
200

ditto, new is not required here

src/main/scala/mesosphere/mesos/TaskGroupBuilder.scala
350 ↗(On Diff #3342)

nit: eliminate this added newline and this file drops from the diff

src/test/scala/mesosphere/marathon/api/v2/AppsResourceTest.scala
168

happy to see some tests here. i'd like a more thorough exercise of the validation rules to appear in AppValidationTest

src/test/scala/mesosphere/marathon/api/v2/PodsResourceTest.scala
240

happy to see some tests here. i'd like a more thorough exercise of the validation rules to appear in PodsValidationTest

zen-dog accepted this revision.May 30 2017, 11:01 AM

Some minor changes but otherwise lgtm. Thx!

tests/system/common.py
875–876

It seems like next 10 lines can be replaced by create_docker_pull_config_json() call?

tests/system/marathon_common_tests.py
986–987

Those asserts here and in test_create_pod_with_private_image() are not necessary since you skip the test if they're not there anyway?

kensipe requested changes to this revision.May 30 2017, 4:22 PM
kensipe added inline comments.
tests/system/common.py
472–482

I don't know what "private_mesos_app" means... can we rename this to something meaningful? or add docs that explains why it should be used?

865

this is in shakedown... all of this can be replaced with shakedown. distribute_docker_credentials_to_private_agents()
https://github.com/dcos/shakedown/blob/master/shakedown/dcos/docker.py#L65

tests/system/marathon_common_tests.py
986–987

I don't see docker_env_set() in the skipif.. however it is likely that we are skipping if we are missing expected ENV.. that makes these assert 'DOCKER_XYZ` not useful.

tests/system/marathon_pods_tests.py
95

do we need these assertions?

110

why do we define the app json for secrets behind a function and for the longer pod json we do not?
I'm a fan of pulling this out so the test focuses on the setup and assertion.. but it isn't that big of a deal.

zen-dog added inline comments.May 30 2017, 4:31 PM
tests/system/common.py
472–482

App id should have a random part (see using of uuid in other tests). Prevents collisions and makes debugging easier.

865

This is not the same - there is no need to distribute the file, since it's passed as part of the app definition. However we should extract creation part into a separate function and maybe move it to shakedown later.

kensipe added inline comments.May 30 2017, 4:42 PM
tests/system/common.py
472–482

your comment doesn't address the comment or concern... the issue isn't app_id... it is the meaning of private_mesos_container and documentation

tests/system/marathon_common_tests.py
986–987

yes

zen-dog added inline comments.May 30 2017, 4:53 PM
tests/system/common.py
472–482

It was meant as an addition to your comment ;)

ichernetsky marked 12 inline comments as done.May 31 2017, 2:28 AM
ichernetsky added inline comments.
tests/system/common.py
865

Aleksey is correct.

875–876

Yep, forgot to do it.

ichernetsky updated this revision to Diff 3356.May 31 2017, 2:28 AM
ichernetsky marked an inline comment as done.
  • Address comments
ichernetsky updated this revision to Diff 3357.May 31 2017, 2:48 AM
  • Use Mesos 1.3.0-rc3
jenkins accepted this revision.May 31 2017, 2:49 AM

✔ Build of 3356 completed jenkins-public-marathon-phabricator-114.

You can create a DC/OS with your patched Marathon by creating a new pull
request with the following changes in buildinfo.json:

"url": "https://downloads.mesosphere.io/marathon/snapshots/marathon-1.5.0-SNAPSHOT-542-gba55703.tgz",
"sha1"" "d1fa8e7e2d01c05830129043c4cdaf3e833dc322"
jenkins added a comment.May 31 2017, 3:02 AM

✔ Build of 3357 completed jenkins-public-marathon-phabricator-115.

You can create a DC/OS with your patched Marathon by creating a new pull
request with the following changes in buildinfo.json:

"url": "https://downloads.mesosphere.io/marathon/snapshots/marathon-1.5.0-SNAPSHOT-542-gba55703.tgz",
"sha1"" "abe173ddd939a9eccadf32115f50264b9e042b9e"
unterstein accepted this revision.May 31 2017, 2:39 PM

wow, a bunch of feedback already given and addressed.
On top I would add only one nit.
Except open feedback LGTM 🚀

Accept in advance

docs/docs/pods.md
184

s/config/pullConfig/

jeschkies resigned from this revision.May 31 2017, 4:07 PM
ichernetsky updated this revision to Diff 3377.May 31 2017, 5:18 PM
  • config -> pullConfig in pods.md
jenkins accepted this revision.May 31 2017, 5:33 PM

✔ Build of 3377 completed jenkins-public-marathon-phabricator-134.

You can create a DC/OS with your patched Marathon by creating a new pull
request with the following changes in buildinfo.json:

"url": "https://downloads.mesosphere.io/marathon/snapshots/marathon-1.5.0-SNAPSHOT-542-gba55703.tgz",
"sha1"" "56551544aaf0dc78a352f40840078fea5255805d"
kensipe accepted this revision.May 31 2017, 5:58 PM
jdef requested changes to this revision.May 31 2017, 9:53 PM

I've left ??? to indicate unanswered questions from prior reviews

docs/docs/rest-api/public/api/v2/types/appContainer.raml
15

???

119

???

docs/docs/rest-api/public/api/v2/types/docker.raml
12

nit: required: true is redundant (because it's the default). i don't think we say that many other places?

src/main/scala/mesosphere/marathon/api/v2/validation/PodsValidation.scala
127

Pretty sure that image.id validation is already handled by the generated RAML code:

val id = json.\("id").validate[String](play.api.libs.json.JsPath.read[String](maxLength[String](1024) keepAnd minLength[String](1)))

so .. no need to duplicate that validation here. the length check above (in the dockerImageValidator) code is redundant, and there's already a JIRA filed for removing redundant validation. we shouldn't add any more

This revision now requires changes to proceed.May 31 2017, 9:53 PM
ichernetsky marked 3 inline comments as done.May 31 2017, 10:14 PM
ichernetsky updated this revision to Diff 3380.May 31 2017, 10:15 PM
  • Address @jdef's comments
jenkins requested changes to this revision.May 31 2017, 10:25 PM
This revision now requires changes to proceed.May 31 2017, 10:25 PM
jenkins added a comment.May 31 2017, 10:25 PM

✗ Build of 3380 failed jenkins-public-marathon-phabricator-140.
Error message:

Stage Compile and Test failed.

jdef added a comment.May 31 2017, 10:35 PM

It'd be pretty sweet if the RAML DockerCredentials type had a deprecation notice somewhere:

DockerCredentials:
  type: object
  description: Credential to authenticate with the docker registry
  properties:
    principal:
      type: string
      description: Principal to authenticate with the docker registry
    secret?:
      type: string
      description: Secret to authenticate with the docker registry
jdef accepted this revision.May 31 2017, 10:36 PM

approving, subject to the addition of the requested deprecation notice

ichernetsky updated this revision to Diff 3382.Jun 1 2017, 12:35 AM
  • Deprecated DockerCredentials type
Closed by commit rMARATHONb97782ca9309: Support passing a Docker config.json for app / pod being launched (authored by ichernetsky). · Explain WhyJun 1 2017, 12:36 AM
This revision was automatically updated to reflect the committed changes.
jenkins added a comment.Jun 1 2017, 12:50 AM

✗ Build of 3382 failed jenkins-public-marathon-phabricator-141.
Error message:

Stage Compile and Test failed.

Revision Contents

PathPackages
Mdocs/docs/blue-green-deploy.md (4 lines)
Mdocs/docs/native-docker.md (55 lines)
Mdocs/docs/pods.md (79 lines)
Mdocs/docs/rest-api/public/api/v2/schema/AppDefinition.json (15 lines)
Mdocs/docs/rest-api/public/api/v2/types/appContainer.raml (9 lines)
AMdocs/docs/rest-api/public/api/v2/types/docker.raml (16 lines)
Mdocs/docs/rest-api/public/api/v2/types/podContainer.raml (6 lines)
Mproject/Dependencies.scala (2 lines)
Mproject/RamlGeneratorPlugin.scala (1 line)
Mproject/plugins.sbt (1 line)
Msrc/main/java/mesosphere/marathon/Protos.java (4543 lines)
Msrc/main/proto/README.md (3 lines)
Msrc/main/proto/marathon.proto (19 lines)
Msrc/main/proto/mesos/mesos.proto (196 lines)
Msrc/main/scala/mesosphere/marathon/api/serialization/ContainerSerializer.scala (43 lines)
AMsrc/main/scala/mesosphere/marathon/api/serialization/SecretSerializer.scala (15 lines)
Msrc/main/scala/mesosphere/marathon/api/v2/validation/AppValidation.scala (17 lines)
Msrc/main/scala/mesosphere/marathon/api/v2/validation/PodsValidation.scala (23 lines)
Msrc/main/scala/mesosphere/marathon/raml/ContainerConversion.scala (34 lines)
Msrc/main/scala/mesosphere/marathon/state/Container.scala (3 lines)
Msrc/test/scala/mesosphere/marathon/api/v2/AppsResourceTest.scala (69 lines)
Msrc/test/scala/mesosphere/marathon/api/v2/PodsResourceTest.scala (149 lines)
Msrc/test/scala/mesosphere/marathon/api/v2/validation/AppValidationTest.scala (52 lines)
Msrc/test/scala/mesosphere/marathon/raml/ContainerConversionTest.scala (18 lines)
Msrc/test/scala/mesosphere/mesos/TaskBuilderTest.scala (7 lines)
Mtests/system/common.py (127 lines)
Mtests/system/marathon_common_tests.py (34 lines)
Mtests/system/marathon_pods_tests.py (29 lines)
Mtests/system/test_marathon_on_marathon_ee.py (2 lines)

Diff 3383

View Options

docs/docs/blue-green-deploy.md

This file was changed only by adding or removing whitespace. Show File Contents
View Options

docs/docs/native-docker.md

Show First 20 Lines • Show All 287 Lines • ▼ Show 20 Line(s)
288288
289289# Mesos Containerizer and Universal Container Runtime
290290
291291 Starting with version 1.3.0, Marathon can provision Docker container images without relying on the external Docker Engine
292292 Instead, the Mesos containerizer uses the Universal Container Runtime (added in [Apache Mesos version 1.0](http://mesos.apache.org/blog/mesos-1-0-0-released/), released July 2016) which uses native OS features to configure and start Docker or [AppC](https://github.com/appc/spec) container images and provide isolation.
293293
294294## Configuration
295295
296Selected this setup by specifying the follow JSON combination, which previously provoked an error message:
297container type "MESOS" and a "docker" object.
296Selected this setup by specifying the following JSON combination, which
297previously provoked an error message: container type `MESOS` and a
298`docker` object.
jdefUnsubmitted
Done

nit: s/follow/following/

298299
299300```json
300301{
301302 "id": "mesos-docker",
302303 "container": {
303304 "docker": {
304305 "image": "mesosphere/inky"
305306 },
306307 "type": "MESOS"
307308 },
308309 "args": ["hello"],
309310 "cpus": 0.2,
310311 "mem": 16.0,
311312 "instances": 1
312313}
313314```
314The Mesos containerizer does not support the same parameter options as the Docker containerizer yet.
315The only properties recognized by both containerizers are "image" and "forcePullImage",
316with the same semantics. All other Docker container properties result in an error with the Mesos containerizer.
317315
318However, the latest version of the Mesos containerizer introduces its own new property, "credential", with a "principal" and an optional "secret" field to authenticate when downloading the Docker image.
316The Mesos containerizer does not support the same parameter options as
317the Docker containerizer yet. The only properties recognized by both
318containerizers are `image` and `forcePullImage`, with the same
319semantics. All other Docker container properties result in an error
320with the Mesos containerizer.
319321
322Starting with Marathon v1.5, Mesos containerizer supports the
jdefUnsubmitted
Done

s/supports/supports the/

323`container.docker.pullConfig` property, which may be used to specify
jdefUnsubmitted
Done

nit: s/might/may/

324a Docker `config.json` per Docker image. Its `secret` property
zen-dogUnsubmitted
Done

Can only be specified as secret to ensure security etc.

325should be set to one of top-level `secrets` properties, which refers to
jdefUnsubmitted
Done

nit: s/field/fields/

326`config.json` in a secret store.
jdefUnsubmitted
Done

Please add a blurb here that clearly indicates that this new method of docker image pull secrets is NOT supported by Docker containerizer; different ways of doing things for different containerizers is typically very confusing for marathon users

jdefUnsubmitted
Done

wordsmith this a bit: "... secret store storing a content". suggest something like:

"... which refers to a secret whose value contains the content of, for example, ~/.docker/config.json."

327
zen-dogUnsubmitted
Done

I would also add a reference to docs/docs/secrets.md (see D729) to make clear that you need to implement you own secrets plugin in order for this to work. Otherwise it's not clear where the secret should come from.

320328```json
321329{
322 "id": "mesos-docker",
330 "id": "/mesos-docker",
323331 "container": {
324332 "docker": {
325333 "image": "mesosphere/inky",
326 "credential": {
327 "principal": "alice",
334 "pullConfig": {
335 "secret": "pullConfig"
jdefUnsubmitted
Done

s/config/pullConfig/

328 "secret": "wonderland"
329336 }
jdefUnsubmitted
Done

ditto, not supported for this release (at least, not in this API)

330337 },
331338 "type": "MESOS"
332339 },
340 "secrets": {
341 "pullConfig": {
342 "source": "/mesos-docker/pullConfig"
343 }
344 },
333345 "args": ["hello"],
334346 "cpus": 0.2,
335347 "mem": 16.0,
336348 "instances": 1
337349}
338350```
339351
zen-dogUnsubmitted
Done

This is just confusing since this way never worked. I would either remove this completely or just say that: "...Previously existing credential object is deprecated..."
Since it was never implemented (so we won't break anything) we should probably just remove it? What do you think @meichstedt, @jdef ?

ichernetskyAuthorUnsubmitted
Not Done

Removed.

352Please note, that this feature is not supported by Docker
353containerizer yet.
354
355To learn more about secrets, please refer to
356[Secrets]({{ site.baseurl/docs/secrets.html }})
357
340358## Resources
341359
342360- [Mesos Docker Containerizer](http://mesos.apache.org/documentation/latest/docker-containerizer)
343- [Supporting Container Images in Mesos Containerizer]
344 (https://github.com/apache/mesos/blob/master/docs/container-image.md)
361- [Supporting Container Images in Mesos Containerizer](https://github.com/apache/mesos/blob/master/docs/container-image.md)
View Options

docs/docs/pods.md

Show First 20 Lines • Show All 69 Lines • ▼ Show 20 Line(s)
7070
7171### Pod Events and State
7272
7373 When you update a pod that has already launched, the new version of the pod will only be available when redeployment is complete. If you query the system to learn which version is deployed before redeployment is complete, you may get the previous version as a response. The same is true for the status of a pod: if you update a pod, the change in status will not be reflected in a query until redeployment is complete.
7474
7575 History is permanently tied to `pod_id`. If you delete a pod and then reuse the ID, even if the details of the pod are different, the new pod will have the previous history (such as version information).
7676
7777### Pod Definitions
78
7879Pods are configured via a JSON pod definition, which is similar to an [application definition]({{ site.baseurl }}/docs/application-basics.html). You must declare the resources required by each container in the pod because Mesos, not Marathon, determines how and when to perform isolation for all resources requested by a pod. See the [Examples](#examples) section for complete pod definitions.
7980
8081#### Executor Resources
8182
8283The executor runs on each node to manage the pods. By default, the executor reserves 32 MB and .1 CPUs per pod for overhead. Take this overhead into account when declaring resource needs for the containers in your pod. You can modify the executor resources in the `executorResources` field of your pod definition.
8384
8485```json
8586{
▲ Show 20 Lines • Show All 68 Lines • ▼ Show 20 Line(s)
154155{
155156 "image":{
156157 "id":"mesosphere/marathon:latest",
157158 "kind":"DOCKER",
158159 "forcePull":false
159160 }
160161}
161162```
163
164An optional `image.pullConfig` is supported too. Here is an example of
jdefUnsubmitted
Done

Please provide a working example here for pods. People really like working examples

165a pod pulling a Docker image from a private registry:
166
167```json
168{
169 "id": "/simple-pod",
170 "scaling": {
171 "kind": "fixed",
172 "instances": 1
173 },
174 "containers": [{
175 "name": "container0",
176 "exec": {
177 "command": {
178 "shell": "sleep 1000"
179 }
180 },
181 "image": {
182 "kind": "DOCKER",
183 "id": "company/private-image",
184 "pullConfig": {
untersteinUnsubmitted
Not Done

s/config/pullConfig/

185 "secret": "configSecret"
186 }
187 },
188 "resources": {
189 "cpus": 1,
190 "mem": 50.0
191 }
192 }],
193 "secrets": {
194 "configSecret": {
195 "source": "/config"
196 }
197 }
198}
199```
200
201For further details, please refer
202to [Configuration of Docker images with Mesos containerizer]({{ site.baseurl }}/docs/native-docker.html).
162203
163204## Create and Manage Pods
164205
165206Use the `/v2/pods/` endpoint to create and manage your pods. [See the full API spec]({{ site.baseurl }}/docs/generated/api.html#v2_pods).
166207
167208### Create
168209
169210```bash
▲ Show 20 Lines • Show All 402 Lines • Show Last 20 Lines
View Options

docs/docs/rest-api/public/api/v2/schema/AppDefinition.json

Show First 20 Lines • Show All 133 Lines • ▼ Show 20 Line(s)
134134 "container": {
135135 "additionalProperties": false,
136136 "properties": {
137137 "docker": {
138138 "additionalProperties": false,
139139 "properties": {
140140 "credential": {
141141 "type": "object",
142 "description": "Credential to authenticate with the docker registry.",
142 "description": "Credential to authenticate with the docker registry. It is deprecated since Marathon v1.5",
jdefUnsubmitted
Done

please add some text indicating that credential is DEPRECATED

143143 "additionalProperties": false,
144144 "properties": {
145145 "principal": {
146146 "type": "string",
147147 "description": "Principal to authenticate with the docker registry."
148148 },
149149 "secret": {
150150 "type": "string",
151151 "description": "Secret to authenticate with the docker registry."
152152 }
153153 },
154154 "required": [ "principal" ]
155 },
156 "pullConfig": {
jdefUnsubmitted
Done

EE? secrets API is OSS

157 "type": "object",
158 "description": "Docker's config.json",
159 "additionalProperties": false,
160 "properties": {
161 "secret" : {
162 "type": "string",
163 "description": "A secret whose value is a stringified JSON object in a Secret Store.",
164 "minLength": 1
165 }
166 },
167 "required": [ "secret" ]
155168 },
156169 "forcePullImage": {
157170 "type": "boolean",
158171 "description": "The container will be pulled, regardless if it is already available on the local system."
159172 },
160173 "image": {
161174 "type": "string",
162175 "minLength": 1,
▲ Show 20 Lines • Show All 669 Lines • Show Last 20 Lines
View Options

docs/docs/rest-api/public/api/v2/types/appContainer.raml

11#%RAML 1.0 Library
22uses:
33 label: label.raml
44 numbers: numberTypes.raml
55 volumes: volumes.raml
66 strings: stringTypes.raml
77 pragma: pragma.raml
8 docker: docker.raml
89
910types:
1011 EngineType:
1112 type: string
1213 enum: [MESOS, DOCKER]
1314 description: |
1415 Container engine type. Supported engine types at the moment are DOCKER and MESOS.
1516 DockerCredentials:
jdefUnsubmitted
Done

this should be deprecated not removed

jdefUnsubmitted
Done

please indicate that this type is deprecated and should no longer be used

jdefUnsubmitted
Not Done

???

jdefUnsubmitted
Not Done

???

1617 type: object
18 (pragma.deprecated): Deprecated since v1.5
1719 description: Credential to authenticate with the docker registry
1820 properties:
1921 principal:
2022 type: string
2123 description: Principal to authenticate with the docker registry
2224 secret?:
2325 type: string
2426 description: Secret to authenticate with the docker registry
▲ Show 20 Lines • Show All 79 Lines • ▼ Show 20 Line(s)
104106 List of the container networks associated with this mapping. If
105107 absent, then this mapping is associated with all defined container
106108 networks (for this application). A single item list is mandatory when
107109 `hostPort` is specified and multiple container networks are defined.
108110 (pragma.omitEmpty):
109111 DockerContainer:
110112 type: object
111113 properties:
112114 credential?:
jdefUnsubmitted
Not Done

add (pragma.deprecated) instead of removing

113115 type: DockerCredentials
114116 description: |
115117 The credentials to fetch this container.
116118 Please note: this property is supported only with the Mesos containerizer, not the docker containerizer.
119 (pragma.deprecated): Deprecated in favor of pullConfig
jdefUnsubmitted
Done

s/config/pullConfig/

jdefUnsubmitted
Done

i'm looking for "Deprecated in favor of pullConfig"

jdefUnsubmitted
Not Done

???

120 pullConfig?:
121 type: docker.DockerPullConfig
122 description: |
jdefUnsubmitted
Done

please add a note here indicating that this field is NOT to be used with the docker containerizer

123 Docker's config.json given as a secret name into a secret store
124 which corresponding value is a content of `~/.docker/config.json`.
125 It is supported only with Mesos containerizer.
jdefUnsubmitted
Done

ditto, API is not EE

117126 forcePullImage?:
118127 type: boolean
119128 description: |
120129 The container will be pulled, regardless if it is already available on
121130 the local system.
122131 default: false
123132 image:
124133 type: string
▲ Show 20 Lines • Show All 62 Lines • Show Last 20 Lines
View Options

docs/docs/rest-api/public/api/v2/types/docker.raml

  • This file was added.
1#%RAML 1.0 Library
2uses:
3 secrets: secrets.raml
4
5types:
6 DockerPullConfig:
7 type: object
jdefUnsubmitted
Done

dislike bike-shedding on names ... but, I'm not crazy about "config" or "DockerConfig". The goal of this item isn't to "configure docker", rather it's to relay information/tokens/credentials/whatever to enable a client to pull an image from a remote server. "config" is too general and easy to abuse (like "util", "common", etc).

"pullToken" / "DockerPullToken" is closer. "Token" might be a bit too specific, but I like it better than the too-general "config". We don't want to imply that users should be editing their docker config files, removing any non-authn fields because who knows how the docker run client impl will actually validate the contents of a docker configuration file at runtime - it might want fields non-essential for pulling images simply because docker isn't taking our use case in mind. So .. in the end I don't like this "token" based name better than config because it's misleading as to the actual (likely) contents of the blob.

"pullConfig" / "ImagePullConfig" is the name best alternative name that I can think of. Though I'm still not crazy about it, I do think it's an improvement over "config" / "DockerConfig" in that it provides clarity w/ regard to intent (pulling images) and to content (yes, it's actually supposed to reference a config blob). Also "ImagePullConfig" would let us reuse this object type for other kinds of image registries that require credentials/tokens/whatever (some future appc image registry that UCR can securely pull images from).

Are there better names out there?

8 properties:
9 secret:
10 type: string
11 minLength: 1
12 description: |
jdefUnsubmitted
Not Done

nit: required: true is redundant (because it's the default). i don't think we say that many other places?

13 Docker's config.json. It should be specifed only as a secret
14 name into a secret store, and the corresponding value should be
jdefUnsubmitted
Done

Product has decided to nix support for DockerConfigText (clear-text docker config in the Marathon API) for now. Please remove.

jdefUnsubmitted
Done

s/can/should/

15 a stringified valid JSON object which can be found in
16 `~/.docker/config.json`.
aquamatthiasUnsubmitted
Not Done

text is arbitrary. Perhaps jsonString ?

ichernetskyAuthorUnsubmitted
Not Done

This is intentional. The idea is to have this property for the time being which value is passed further as is, and in the meantime in addition to it implement json one. How does it sound to you?

View Options

docs/docs/rest-api/public/api/v2/types/podContainer.raml

11#%RAML 1.0 Library
22uses:
33 artifacts: artifact.raml
44 env: environmentVariable.raml
55 health: healthCheck.raml
66 labels: label.raml
77 mesosCommand: mesosCommand.raml
88 network: network.raml
99 pragma: pragma.raml
1010 resources: resources.raml
1111 strings: stringTypes.raml
1212 volumes: volumes.raml
1313 containerCommons: containerCommons.raml
14 docker: docker.raml
1415
1516types:
1617 ImageType:
1718 type: string
1819 enum: [DOCKER, APPC]
1920 Image:
2021 type: object
2122 properties:
2223 kind: ImageType
2324 id:
2425 type: strings.Path
2526 description: address/identifier of the file system image
2627 example: mesosphere/marathon:1.3.0
28 pullConfig?:
jdefUnsubmitted
Done

don't need to worry about a note for docker containerizer vs. mesos containerizer here since pods only support UCR

29 type: docker.DockerPullConfig
30 description: |
31 Name of a secret whose value contains a stringified
jdefUnsubmitted
Done

wordsmith this a bit: "Name of a secret whose value contains a stringified Docker config.json" ?

32 Docker config.json
timcharperUnsubmitted
Not Done

Is this the best way to word it? Secret store for Marathon is pluggable

ichernetskyAuthorUnsubmitted
Not Done

Would "tool to manage secrets" be better?

2733 forcePull?:
2834 type: boolean
2935 description: true if the image should always be pulled
3036 labels?:
3137 type: labels.KVLabels
3238 (pragma.omitEmpty):
3339 description: |
3440 Used during image discovery and dependency resolution.
▲ Show 20 Lines • Show All 81 Lines • Show Last 20 Lines
View Options

project/Dependencies.scala

Show First 20 Lines • Show All 77 Lines • ▼ Show 20 Line(s)
7878
7979object Dependency {
8080 object V {
8181 // runtime deps versions
8282 val Aws = "1.11.128"
8383 val Alpakka = "0.8"
8484 val Chaos = "0.8.8"
8585 val Guava = "19.0"
86 val Mesos = "1.2.0"
86 val Mesos = "1.3.0-rc3"
jeschkiesUnsubmitted
Not Done

Is this correct? Are we jumping to Mesos 1.4?

ichernetskyAuthorUnsubmitted
Not Done

Nope, it is just the protobufs in the latest 1.3.0-SNAPSHOT are compiled with protoc 3.3.0, which causes a lot of errors and warnings, whereas the latest build from master branch has everything we need for this feature, and the protobufs are compiled with protoc 2.6.1

jdefUnsubmitted
Not Done

is the plan to merge this changeset with the 1.4.0 dependency like this? I hope not. But if it is, then let's make sure we have a JIRA tracker to fix this value before we release (please link to the JIRA here and in the top-level summary)

jdefUnsubmitted
Not Done

this should really be fixed before merging this changeset

ichernetskyAuthorUnsubmitted
Not Done

Unfortunately, there is some experimentation with protobuf versions in 1.3 branch, but since protobuf 2.6.1 is still used in master, and it works just fine, 1.4.0-SNAPSHOT is used here. It will be adjusted before closing the current ticket, so that there is no need to create a new one.

jdefUnsubmitted
Not Done

I've heard that there's a 1.3.0-rc1 that contains what we need?

8787 // Version of Mesos to use in Dockerfile.
8888 val MesosDebian = "1.2.0-2.0.6"
8989 val OpenJDK = "openjdk:8u121-jdk"
9090 val Akka = "2.4.18"
9191 val AkkaHttp = "10.0.6"
9292 val AkkaSSE = "2.0.0"
9393 val ApacheCommonsCompress = "1.13"
9494 val ApacheCommonsIO = "2.4"
▲ Show 20 Lines • Show All 102 Lines • Show Last 20 Lines
View Options

project/RamlGeneratorPlugin.scala

11package mesosphere.raml
22
3import java.io.{FileOutputStream, ObjectOutputStream}
43import java.nio.charset.StandardCharsets
54import java.security.MessageDigest
65import java.util.Base64
76
87import sbt._
98import sbt.Keys._
109import org.raml.v2.api.RamlModelBuilder
1110import org.yaml.snakeyaml.Yaml
▲ Show 20 Lines • Show All 129 Lines • Show Last 20 Lines
View Options

project/plugins.sbt

Show All 16 Lines
1717addSbtPlugin("com.typesafe.sbt" % "sbt-git" % "0.8.5")
1818addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.1.5")
1919addSbtPlugin("com.sksamuel.scapegoat" %% "sbt-scapegoat" % "1.0.4")
2020addSbtPlugin("de.johoop" % "cpd4sbt" % "1.2.0")
2121addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.2.15")
2222addSbtPlugin("com.typesafe.sbt" % "sbt-aspectj" % "0.10.6")
2323addSbtPlugin("io.get-coursier" % "sbt-coursier" % "1.0.0-M15")
2424
25
2625libraryDependencies ++= Seq(
2726 "org.raml" % "raml-parser-2" % "1.0.0",
2827 "com.eed3si9n" %% "treehugger" % "0.4.1",
2928 "org.slf4j" % "slf4j-nop" % "1.7.22",
3029 "org.vafer" % "jdeb" % "1.3" artifacts Artifact("jdeb", "jar", "jar"),
3130 "com.typesafe.play" %% "play-json" % "2.4.11"
3231)
3332
3433sbtPlugin := true
3534
View Options

src/main/java/mesosphere/marathon/Protos.java

This file has a very large number of changes (4,543 lines). Show File Contents
View Options

src/main/proto/README.md

11# Google Protocol Buffers
22
33Marathon is using Google [Protocol Buffers](https://developers.google.com/protocol-buffers) to marshal and unmarshal data from the persistent store.
44
55## Dependencies
66
7- Google [Protocol Buffers](https://developers.google.com/protocol-buffers) v2.5 should be installed on the system.
7- Google [Protocol Buffers](https://developers.google.com/protocol-buffers) v2.6.1 should be installed on the system.
88- The `src/main/proto/mesos/mesos.proto` should correspond to the [Mesos](http://mesos.apache.org) library
99 version.
1010
1111## Rebuilt the protos
1212
1313To rebuild the protos, open a terminal where the current working directory is the project directory.
1414
1515```
1616$> cd src/main/proto
1717$> protoc --java_out=../java/ marathon.proto
1818```
19
View Options

src/main/proto/marathon.proto

Show First 20 Lines • Show All 226 Lines • ▼ Show 20 Line(s)
227227}
228228
229229// Copied from mesos.proto and adjusted to meet the needs for persistent volumes
230230// Extends DockerInfo.PortMapping to include `service_port`.
231231message ExtendedContainerInfo {
232232
233233 // Docker on a Docker engine
234234 message DockerInfo {
235 message ImagePullConfig {
236 enum Type {
237 SECRET = 1;
238 }
235239
240 required Type type = 1;
241
242 // The UTF-8 character encoded byte data, which is expected as
243 // a docker config file in JSON format. This field is used for
244 // supporting docker private registry credential per container.
245 // Users can specify different docker config files for pulling
246 // their private images from different registries.
247 optional mesos.Secret secret = 2;
248 }
249
236250 required string image = 1;
237251
238252 // deprecated in favor of ServiceDefinition.networks since 1.5
239253 optional mesos.ContainerInfo.DockerInfo.Network OBSOLETE_network = 2 [default = HOST];
240254
241255 // deprecated in favor of ExtendedContainerInfo.PortMapping,since 1.5
242256 message ObsoleteDockerPortMapping {
243257 optional uint32 host_port = 1;
Show All 27 Lines
271285 // [REGISTRY_HOST[:REGISTRY_PORT]/]REPOSITORY[:TAG|@TYPE:DIGEST]
272286 //
273287 // See: https://docs.docker.com/reference/commandline/pull/
274288 required string image = 1;
275289
276290 // Credential to authenticate with docker registry.
277291 // NOTE: This is not encrypted, therefore framework and operators
278292 // should enable SSL when passing this information.
279 optional mesos.Credential credential = 2;
293 optional mesos.Credential deprecated_credential = 2; // Deprecated since 1.5
jdefUnsubmitted
Done

it's probably not obvious or written anywhere in our documentation, but the way we've traditionally deprecated proto fields in marathon was by actually changing the name of the field. in this case it would be called deprecated_credential. and the comment would include a note like "Deprecated since 1.5"

When we finally stop reading/writing the field (except for migrations) then the deprecated_ prefix changes to OBSOLETE_ (or something like that)

jdefUnsubmitted
Done

please include a comment "deprecated since 1.5"

280294
281295 // With this flag set to true, the universal Mesos containerizer will
282296 // pull the docker image from the registry even if the image
283297 // is already downloaded on the agent.
284298 optional bool force_pull_image = 3;
299
300 // Docker config.json which is used when pulling Docker images.
301 optional DockerInfo.ImagePullConfig pull_config = 4;
285302 }
286303
287304 // AppC on the universal Mesos engine
288305 message MesosAppCInfo {
jdefUnsubmitted
Done

two points:

(1) ditto w/ respect to bike-shedding on the same. pull_config is probably more clear, but I'm open to suggestions

(2) please create a wrapper for this instead of assuming that image pull config will always be a secret, something like:

message ImagePullConfig {
  optional mesos.Secret secret = 1;
}

...
optional ImagePullConfig pull_config = 4;
289306 // The name of the image.
290307 required string image = 1;
291308
292309 // An image ID is a string of the format "hash-value", where
293310 // "hash" is the hash algorithm used and "value" is the hex
294311 // encoded string of the digest. Currently the only permitted
295312 // hash algorithm is sha512.
296313 optional string id = 2;
▲ Show 20 Lines • Show All 177 Lines • Show Last 20 Lines
View Options

src/main/proto/mesos/mesos.proto

11// Licensed to the Apache Software Foundation (ASF) under one
22// or more contributor license agreements. See the NOTICE file
33// distributed with this work for additional information
44// regarding copyright ownership. The ASF licenses this file
55// to you under the Apache License, Version 2.0 (the
66// "License"); you may not use this file except in compliance
77// with the License. You may obtain a copy of the License at
88//
99// http://www.apache.org/licenses/LICENSE-2.0
1010//
1111// Unless required by applicable law or agreed to in writing, software
1212// distributed under the License is distributed on an "AS IS" BASIS,
1313// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1414// See the License for the specific language governing permissions and
1515// limitations under the License.
1616
17syntax = "proto2";
jdefUnsubmitted
Not Done

what version of mesos is this file from? I hope it's the same version (1.3) that we intend to launch with on DC/OS 1.10.

If that's not the case, why? (and ... can we have a JIRA (if needed) to track the task of actually using the the mesos-1.3.0 protos?)

jdefUnsubmitted
Not Done

??

ichernetskyAuthorUnsubmitted
Not Done

It is from 1.3 branch, but it is (was) identical to the one in master at the time.

18
1719package mesos;
1820
1921option java_package = "org.apache.mesos";
2022option java_outer_classname = "Protos";
2123
2224
2325/**
2426 * Status is used to indicate the state of the scheduler and executor
▲ Show 20 Lines • Show All 64 Lines • ▼ Show 20 Line(s)
8991 */
9092message ContainerID {
9193 required string value = 1;
9294 optional ContainerID parent = 2;
9395}
9496
9597
9698/**
99 * A unique ID assigned to a resource provider. Currently, a resource
100 * provider gets a new ID whenever it (re)registers with Mesos.
101 */
102message ResourceProviderID {
103 required string value = 1;
104}
105
106
107/**
97108 * Represents time since the epoch, in nanoseconds.
98109 */
99110message TimeInfo {
100111 required int64 nanoseconds = 1;
101
102 // TODO(josephw): Add time zone information, if necessary.
103112}
104113
105114
106115/**
107116 * Represents duration in nanoseconds.
108117 */
109118message DurationInfo {
110119 required int64 nanoseconds = 1;
▲ Show 20 Lines • Show All 137 Lines • ▼ Show 20 Line(s)
248257 // Roles are the entities to which allocations are made.
249258 // The framework must have at least one role in order to
250259 // be offered resources. Note that `role` is deprecated
251260 // in favor of `roles` and only one of these fields must
252261 // be used. Since we cannot distinguish between empty
253262 // `roles` and the default unset `role`, we require that
254263 // frameworks set the `MULTI_ROLE` capability if
255264 // setting the `roles` field.
256 //
257 // NOTE: The implmentation for supporting `roles`
258 // is not complete, DO NOT USE the `roles` field.
259265 optional string role = 6 [default = "*", deprecated=true];
260 repeated string roles = 12; // EXPERIMENTAL.
266 repeated string roles = 12;
jdefUnsubmitted
Not Done

oooh, this is coming in mesos 1.3?

ichernetskyAuthorUnsubmitted
Not Done

Yes. https://github.com/apache/mesos/blob/1.3.x/include/mesos/mesos.proto#L266

261267
262268 // Used to indicate the current host from which the scheduler is
263269 // registered in the Mesos Web UI. If set to an empty string Mesos
264270 // will automagically set it to the current hostname if one is
265271 // available.
266272 optional string hostname = 7;
267273
268274 // This field should match the credential's principal the framework
▲ Show 20 Lines • Show All 98 Lines • ▼ Show 20 Line(s)
367373 *
368374 * NOTE: This API is unstable and the related feature is experimental.
369375 */
370376message CheckInfo {
371377 enum Type {
372378 UNKNOWN = 0;
373379 COMMAND = 1;
374380 HTTP = 2;
381 TCP = 3;
375382
376 // TODO(alexr): Consider supporting TCP checks and custom user checks.
377 // The latter should probably be paired with a `data` field and
378 // complemented by a `data` response in `CheckStatusInfo`.
383 // TODO(alexr): Consider supporting custom user checks. They should
384 // probably be paired with a `data` field and complemented by a
385 // `data` response in `CheckStatusInfo`.
379386 }
380387
381388 // Describes a command check. If applicable, enters mount and/or network
382389 // namespaces of the task.
383390 message Command {
384391 required CommandInfo command = 1;
385392 }
386393
Show All 12 Lines
399406
400407 // TODO(alexr): Add support for custom HTTP headers.
401408
402409 // TODO(alexr): Consider adding an optional message to describe TLS
403410 // options and thus enabling https. Such message might contain certificate
404411 // validation, TLS version.
405412 }
406413
414 // Describes a TCP check, i.e. based on establishing a TCP connection to
415 // the specified port. Note that <host> is not configurable and is resolved
416 // automatically to 127.0.0.1.
417 message Tcp {
418 required uint32 port = 1;
419 }
420
407421 // The type of the check.
408422 optional Type type = 1;
409423
410424 // Command check.
411425 optional Command command = 2;
412426
413427 // HTTP check.
414428 optional Http http = 3;
415429
430 // TCP check.
431 optional Tcp tcp = 7;
432
416433 // Amount of time to wait to start checking the task after it
417434 // transitions to `TASK_RUNNING` or `TASK_STARTING` if the latter
418435 // is used by the executor.
419436 optional double delay_seconds = 4 [default = 15.0];
420437
421438 // Interval between check attempts, i.e., amount of time to wait after
422439 // the previous check finished or timed out to start the next check.
423440 optional double interval_seconds = 5 [default = 10.0];
▲ Show 20 Lines • Show All 438 Lines • ▼ Show 20 Line(s)
862879 optional Value.Scalar scalar = 3;
863880 optional Value.Ranges ranges = 4;
864881 optional Value.Set set = 6;
865882 optional Value.Text text = 5;
866883}
867884
868885
869886/**
870 * Describes a resource on a machine. The `name` field is a string
871 * like "cpus" or "mem" that indicates which kind of resource this is;
872 * the rest of the fields describe the properties of the resource. A
873 * resource can take on one of three types: scalar (double), a list of
874 * finite and discrete ranges (e.g., [1-10, 20-30]), or a set of
875 * items. A resource is described using the standard protocol buffer
876 * "union" trick.
887 * Describes a resource from a resource provider. The `name` field is
888 * a string like "cpus" or "mem" that indicates which kind of resource
889 * this is; the rest of the fields describe the properties of the
890 * resource. A resource can take on one of three types: scalar
891 * (double), a list of finite and discrete ranges (e.g., [1-10,
892 * 20-30]), or a set of items. A resource is described using the
893 * standard protocol buffer "union" trick.
877894 *
878895 * Note that "disk" and "mem" resources are scalar values expressed in
879896 * megabytes. Fractional "cpus" values are allowed (e.g., "0.5"),
880897 * which correspond to partial shares of a CPU.
881898 */
882899message Resource {
900 optional ResourceProviderID provider_id = 12;
901
883902 required string name = 1;
884903 required Value.Type type = 2;
885904 optional Value.Scalar scalar = 3;
886905 optional Value.Ranges ranges = 4;
887906 optional Value.Set set = 5;
888907
889908 // The role that this resource is reserved for. If "*", this indicates
890909 // that the resource is unreserved. Otherwise, the resource will only
▲ Show 20 Lines • Show All 87 Lines • ▼ Show 20 Line(s)
978997 // task/executor terminates. Currently, if 'persistence' is set,
979998 // 'volume' must be set.
980999 optional Volume volume = 2;
9811000
9821001 // Describes where a disk originates from.
9831002 // TODO(jmlvanre): Add support for BLOCK devices.
9841003 message Source {
9851004 enum Type {
1005 UNKNOWN = 0;
9861006 PATH = 1;
9871007 MOUNT = 2;
9881008 }
9891009
9901010 // A folder that can be located on a separate disk device. This
9911011 // can be shared and carved up as necessary between frameworks.
9921012 message Path {
9931013 // Path to the folder (e.g., /mnt/raid/disk0).
994 required string root = 1;
1014 optional string root = 1;
timcharperUnsubmitted
Not Done

can you help me understand why this was changed? These fields are definitely not optional in the context for which they're used :)

ichernetskyAuthorUnsubmitted
Not Done

I have no idea. It was changed in upstream (mesos.proto)

jdefUnsubmitted
Not Done

changed in preparation of the upcoming support for mesos local/external resource providers and initial CSI support: https://github.com/apache/mesos/commit/5660d7633280bf592b12eeed88c4ebadb33f44c5

9951015 }
9961016
9971017 // A mounted file-system set up by the Agent administrator. This
9981018 // can only be used exclusively: a framework cannot accept a
9991019 // partial amount of this disk.
10001020 message Mount {
10011021 // Path to mount point (e.g., /mnt/raid/disk0).
1002 required string root = 1;
1022 optional string root = 1;
timcharperUnsubmitted
Done

ditto :)

10031023 }
10041024
10051025 required Type type = 1;
10061026 optional Path path = 2;
10071027 optional Mount mount = 3;
10081028 }
10091029
10101030 optional Source source = 3;
Show All 16 Lines
10271047
10281048 // If this is set, the resources are shared, i.e. multiple tasks
10291049 // can be launched using this resource and all of them shall refer
10301050 // to the same physical resource on the cluster. Note that only
10311051 // persistent volumes can be shared currently.
10321052 optional SharedInfo shared = 10;
10331053}
10341054
1055
10351056/**
10361057 * When the network bandwidth caps are enabled and the container
10371058 * is over its limit, outbound packets may be either delayed or
10381059 * dropped completely either because it exceeds the maximum bandwidth
10391060 * allocation for a single container (the cap) or because the combined
10401061 * network traffic of multiple containers on the host exceeds the
10411062 * transmit capacity of the host (the share). We can report the
10421063 * following statistics for each of these conditions exported directly
▲ Show 20 Lines • Show All 116 Lines • ▼ Show 20 Line(s)
11591180message SNMPStatistics {
11601181 optional IpStatistics ip_stats = 1;
11611182 optional IcmpStatistics icmp_stats = 2;
11621183 optional TcpStatistics tcp_stats = 3;
11631184 optional UdpStatistics udp_stats = 4;
11641185}
11651186
11661187
1188message DiskStatistics {
1189 optional Resource.DiskInfo.Source source = 1;
1190 optional Resource.DiskInfo.Persistence persistence = 2;
1191 optional uint64 limit_bytes = 3;
1192 optional uint64 used_bytes = 4;
1193}
1194
1195
11671196/**
11681197 * A snapshot of resource usage statistics.
11691198 */
11701199message ResourceStatistics {
11711200 required double timestamp = 1; // Snapshot time, in seconds since the Epoch.
11721201
11731202 optional uint32 processes = 30;
11741203 optional uint32 threads = 31;
▲ Show 20 Lines • Show All 58 Lines • ▼ Show 20 Line(s)
12331262 optional uint64 mem_low_pressure_counter = 32;
12341263 optional uint64 mem_medium_pressure_counter = 33;
12351264 optional uint64 mem_critical_pressure_counter = 34;
12361265
12371266 // Disk Usage Information for executor working directory.
12381267 optional uint64 disk_limit_bytes = 26;
12391268 optional uint64 disk_used_bytes = 27;
12401269
1270 // Per disk (resource) statistics.
1271 repeated DiskStatistics disk_statistics = 43;
1272
12411273 // Perf statistics.
12421274 optional PerfStatistics perf = 13;
12431275
12441276 // Network Usage Information:
12451277 optional uint64 net_rx_packets = 14;
12461278 optional uint64 net_rx_bytes = 15;
12471279 optional uint64 net_rx_errors = 16;
12481280 optional uint64 net_rx_dropped = 17;
▲ Show 20 Lines • Show All 482 Lines • ▼ Show 20 Line(s)
17311763* `command` or `http` must be set. If the result of the check is not available
17321764* (e.g., the check timed out), these fields must contain empty messages, i.e.,
17331765* `exit_code` or `status_code` will be unset.
17341766*
17351767* NOTE: This API is unstable and the related feature is experimental.
17361768*/
17371769message CheckStatusInfo {
17381770 message Command {
1739 // Exit code of a command check.
1771 // Exit code of a command check. It is the result of calling
1772 // `WEXITSTATUS()` on `waitpid()` termination information on
1773 // Posix and calling `GetExitCodeProcess()` on Windows.
17401774 optional int32 exit_code = 1;
17411775 }
17421776
17431777 message Http {
17441778 // HTTP status code of an HTTP check.
17451779 optional uint32 status_code = 1;
17461780 }
17471781
1782 message Tcp {
1783 // Whether a TCP connection succeeded.
1784 optional bool succeeded = 1;
1785 }
1786
17481787 // TODO(alexr): Consider adding a `data` field, which can contain, e.g.,
17491788 // truncated stdout/stderr output for command checks or HTTP response body
17501789 // for HTTP checks. Alternatively, it can be an even shorter `message` field
17511790 // containing the last line of stdout or Reason-Phrase of the status line of
17521791 // the HTTP response.
17531792
17541793 // The type of the check this status corresponds to.
17551794 optional CheckInfo.Type type = 1;
17561795
17571796 // Status of a command check.
17581797 optional Command command = 2;
17591798
17601799 // Status of an HTTP check.
17611800 optional Http http = 3;
17621801
1802 // Status of a TCP check.
1803 optional Tcp tcp = 4;
1804
17631805 // TODO(alexr): Consider introducing a "last changed at" timestamp, since
17641806 // task status update's timestamp may not correspond to the last check's
17651807 // state, e.g., for reconciliation.
17661808
17671809 // TODO(alexr): Consider introducing a `reason` enum here to explicitly
17681810 // distinguish between completed, delayed, and timed out checks.
17691811}
17701812
▲ Show 20 Lines • Show All 120 Lines • ▼ Show 20 Line(s)
18911933 // includes unused resources sooner or later than the default).
18921934 optional double refuse_seconds = 1 [default = 5.0];
18931935}
18941936
18951937
18961938/**
18971939* Describes a collection of environment variables. This is used with
18981940* CommandInfo in order to set environment variables before running a
1899* command.
1941* command. The contents of each variable may be specified as a string
1942* or a Secret; only one of `value` and `secret` must be set.
19001943*/
19011944message Environment {
19021945 message Variable {
19031946 required string name = 1;
1904 // NOTE: The `value` field was made optional in Mesos 1.2 but it
1905 // is currently enforced to be set. This constraint will be
1906 // removed in a future version.
1947
1948 enum Type {
1949 UNKNOWN = 0;
1950 VALUE = 1;
1951 SECRET = 2;
1952 }
1953
1954 // In Mesos 1.2, the `Environment.variables.value` message was made
1955 // optional. The default type for `Environment.variables.type` is now VALUE,
1956 // which requires `value` to be set, maintaining backward compatibility.
1957 //
1958 // TODO(greggomann): The default can be removed in Mesos 2.1 (MESOS-7134).
1959 optional Type type = 3 [default = VALUE];
1960
1961 // Only one of `value` and `secret` must be set.
19071962 optional string value = 2;
1963 optional Secret secret = 4;
19081964 }
19091965
19101966 repeated Variable variables = 1;
19111967}
19121968
19131969
19141970/**
19151971 * A generic (key, value) pair used in various places for parameters.
Show All 33 Lines
19492005 * 'principal' and 'secret' respectively), etc.
19502006 */
19512007message Credentials {
19522008 repeated Credential credentials = 1;
19532009}
19542010
19552011
19562012/**
2013 * Secret used to pass privileged information. It is designed to provide
2014 * pass-by-value or pass-by-reference semantics, where the REFERENCE type can be
2015 * used by custom modules which interact with a secure back-end.
2016 */
2017message Secret
2018{
2019 enum Type {
2020 UNKNOWN = 0;
2021 REFERENCE = 1;
2022 VALUE = 2;
2023 }
2024
2025 // Can be used by modules to refer to a secret stored in a secure back-end.
2026 // The `key` field is provided to permit reference to a single value within a
2027 // secret containing arbitrary key-value pairs.
2028 //
2029 // For example, given a back-end secret store with a secret named
2030 // "my-secret" containing the following key-value pairs:
2031 //
2032 // {
2033 // "username": "my-user",
2034 // "password": "my-password
2035 // }
2036 //
2037 // the username could be referred to in a `Secret` by specifying
2038 // "my-secret" for the `name` and "username" for the `key`.
2039 message Reference
2040 {
2041 required string name = 1;
2042 optional string key = 2;
2043 }
2044
2045 // Used to pass the value of a secret.
2046 message Value
2047 {
2048 required bytes data = 1;
2049 }
2050
2051 optional Type type = 1;
2052
2053 // Only one of `reference` and `value` must be set.
2054 optional Reference reference = 2;
2055 optional Value value = 3;
2056}
2057
2058
2059/**
19572060 * Rate (queries per second, QPS) limit for messages from a framework to master.
19582061 * Strictly speaking they are the combined rate from all frameworks of the same
19592062 * principal.
19602063 */
19612064message RateLimit {
19622065 // Leaving QPS unset gives it unlimited rate (i.e., not throttled),
19632066 // which also implies unlimited capacity.
19642067 optional double qps = 1;
▲ Show 20 Lines • Show All 63 Lines • ▼ Show 20 Line(s)
20282131 // [REGISTRY_HOST[:REGISTRY_PORT]/]REPOSITORY[:TAG|@TYPE:DIGEST]
20292132 //
20302133 // See: https://docs.docker.com/reference/commandline/pull/
20312134 required string name = 1;
20322135
20332136 // Credential to authenticate with docker registry.
20342137 // NOTE: This is not encrypted, therefore framework and operators
20352138 // should enable SSL when passing this information.
2036 optional Credential credential = 2;
2139 //
2140 // This is field is never used in Mesos before and is deprecated
2141 // since Mesos 1.3. Please use the `Secret::Value` field `config`
2142 // below (see MESOS-7088 for details).
2143 optional Credential credential = 2 [deprecated = true]; // Since 1.3.
2144
2145 // The UTF-8 character encoded byte data, which is expected as
2146 // a docker config file in JSON format. This field is used for
2147 // supporting docker private registry credential per container.
2148 // Users can specify different docker config files for pulling
2149 // their private images from different registries.
2150 optional Secret config = 3;
20372151 }
20382152
20392153 required Type type = 1;
20402154
20412155 // Only one of the following image messages should be set to match
20422156 // the type.
20432157 optional Appc appc = 2;
20442158 optional Docker docker = 3;
▲ Show 20 Lines • Show All 42 Lines • ▼ Show 20 Line(s)
20872201 // ensure that if 'type' is not set, the default value
20882202 // is UNKNOWN. This enables enum values to be added
20892203 // in a backwards-compatible way. See: MESOS-4997.
20902204 UNKNOWN = 0;
20912205
20922206 // TODO(gyliu513): Add HOST_PATH and IMAGE as volume source type.
20932207 DOCKER_VOLUME = 1;
20942208 SANDBOX_PATH = 2;
2209 SECRET = 3;
20952210 }
20962211
20972212 message DockerVolume {
20982213 // Driver of the volume, it can be flocker, convoy, raxrey etc.
20992214 optional string driver = 1;
21002215
21012216 // Name of the volume.
21022217 required string name = 2;
Show All 26 Lines
21292244
21302245 // The following specifies the source of this volume. At most one of
21312246 // the following should be set.
21322247
21332248 // The source of the volume created by docker volume driver.
21342249 optional DockerVolume docker_volume = 2;
21352250
21362251 optional SandboxPath sandbox_path = 3;
2252
2253 // The volume/secret isolator uses the secret-fetcher module (third-party or
2254 // internal) downloads the secret and makes it available at container_path.
2255 optional Secret secret = 4;
21372256 }
21382257
21392258 optional Source source = 5;
21402259}
21412260
21422261
21432262/**
21442263 * Describes a network request from a framework as well as network resolution
▲ Show 20 Lines • Show All 482 Lines • ▼ Show 20 Line(s)
26272746 // definition on all systems and is portable.
26282747 optional uint32 mode = 5;
26292748
26302749 // User ID of owner.
26312750 optional string uid = 6;
26322751
26332752 // Group ID of owner.
26342753 optional string gid = 7;
2754}
2755
2756
2757/**
2758 * Describes information abount a device.
2759 */
2760message Device {
2761 required string path = 1;
2762}
2763
2764
2765/**
2766 * Describes a device whitelist entry that expose from host to container.
2767 */
2768message DeviceAccess {
2769 message Access {
2770 optional bool read = 1;
2771 optional bool write = 2;
2772 optional bool mknod = 3;
2773 }
2774 required Device device = 1;
2775 required Access access = 2;
2776}
2777
2778
2779message DeviceWhitelist {
2780 repeated DeviceAccess allowed_devices = 1;
26352781}
View Options

src/main/scala/mesosphere/marathon/api/serialization/ContainerSerializer.scala

Show First 20 Lines • Show All 319 Lines • ▼ Show 20 Line(s)
320320 def toMesos(credential: Container.Credential): mesos.Protos.Credential = {
321321 val builder = mesos.Protos.Credential.newBuilder
322322 .setPrincipal(credential.principal)
323323 credential.secret.foreach(builder.setSecret)
324324 builder.build
325325 }
326326}
327327
328object DockerPullConfigSerializer {
329 def fromProto(pullConfig: Protos.ExtendedContainerInfo.DockerInfo.ImagePullConfig): Container.DockerPullConfig = {
330 pullConfig.when(_.getType == Protos.ExtendedContainerInfo.DockerInfo.ImagePullConfig.Type.SECRET, _ => {
timcharperUnsubmitted
Done

have you seen mesosphere.mesos.protos.Implicits ?

you can write

secret.when(_.hasType, _.getType).flatMap {
  case mesos.Protos.Secret.Type.VALUE =>
    secret.when(_.hasValue, _.getValue.getData.toStringUtf8)
  case mesos.Protos.Secret.Type.REFERENCE =>
    secret.when(_.hasReference _.getReference.getName)
}.map(Container.DockerConfig(_))

What you wrote is fine and has certainly has less allocations :) Might be worth considering, though, if you like it.

See NetworkConversion.scala for more examples

ichernetskyAuthorUnsubmitted
Not Done

Thanks, Tim. Yes, I saw it, but frankly didn't memorize everything in it.

331 pullConfig.when(_.hasSecret, _.getSecret).flatMap { secret =>
332 secret.when(_.hasType, _.getType).flatMap {
333 case mesos.Protos.Secret.Type.REFERENCE =>
zen-dogUnsubmitted
Done

Only secret references in 1.10

jdefUnsubmitted
Done

I'd be tempted to throw a SerializationFailedException here instead of swallowing unsupported types:

case unsupported => throw SerializationFailedException(s"unsupported secret type: $unsupported")
334 secret.when(_.hasReference, _.getReference.getName).map(Container.DockerPullConfig)
335 case _ => None
336 }
337 }
338 }).flatten match {
339 case Some(deserializedPullConfig) => deserializedPullConfig
340 case _ =>
341 throw SerializationFailedException(s"Failed to deserialize a docker pull config: $pullConfig")
jdefUnsubmitted
Done

nit: should not need new here

342 }
343 }
zen-dogUnsubmitted
Done

Same here and below...

344
345 def toProto(pullConfig: Container.DockerPullConfig): Protos.ExtendedContainerInfo.DockerInfo.ImagePullConfig = pullConfig match {
346 case Container.DockerPullConfig(secret) =>
347 val builder = Protos.ExtendedContainerInfo.DockerInfo.ImagePullConfig.newBuilder
348 builder.setType(Protos.ExtendedContainerInfo.DockerInfo.ImagePullConfig.Type.SECRET)
349 val secretProto = SecretSerializer.toSecretReference(secret)
350 builder.setSecret(secretProto)
351 builder.build
352 }
353
354 def toMesos(pullConfig: Container.DockerPullConfig): mesos.Protos.Secret = pullConfig match {
355 case Container.DockerPullConfig(secret) =>
356 SecretSerializer.toSecretReference(secret)
357 }
358}
359
328360object MesosDockerSerializer {
329361 def fromProto(proto: Protos.ExtendedContainerInfo): Container.MesosDocker = {
330362 val d = proto.getMesosDocker
331363 val pms = proto.getPortMappingsList
332364 Container.MesosDocker(
333365 volumes = proto.getVolumesList.map(Volume(_))(collection.breakOut),
334366 portMappings = pms.map(PortMappingSerializer.fromProto)(collection.breakOut),
335367 image = d.getImage,
336 credential = if (d.hasCredential) Some(CredentialSerializer.fromMesos(d.getCredential)) else None,
368 credential = if (d.hasDeprecatedCredential) Some(CredentialSerializer.fromMesos(d.getDeprecatedCredential)) else None,
369 pullConfig = if (d.hasPullConfig) Some(DockerPullConfigSerializer.fromProto(d.getPullConfig)) else None,
337370 forcePullImage = if (d.hasForcePullImage) d.getForcePullImage else false
338371 )
339372 }
340373
341374 def toProto(docker: Container.MesosDocker): Protos.ExtendedContainerInfo.MesosDockerInfo = {
342375 val builder = Protos.ExtendedContainerInfo.MesosDockerInfo.newBuilder
343376 .setImage(docker.image)
344377 .setForcePullImage(docker.forcePullImage)
345378
346379 docker.credential.foreach { credential =>
347 builder.setCredential(CredentialSerializer.toMesos(credential))
380 builder.setDeprecatedCredential(CredentialSerializer.toMesos(credential))
348381 }
382 docker.pullConfig.foreach { pullConfig =>
383 builder.setPullConfig(DockerPullConfigSerializer.toProto(pullConfig))
384 }
349385
350386 builder.build
351387 }
352388
353389 def toMesos(container: Container.MesosDocker): mesos.Protos.ContainerInfo.MesosInfo = {
354390 val dockerBuilder = mesos.Protos.Image.Docker.newBuilder
355391 .setName(container.image)
356392
357393 container.credential.foreach { credential =>
358394 dockerBuilder.setCredential(CredentialSerializer.toMesos(credential))
395 }
396 container.pullConfig.foreach { pullConfig =>
397 dockerBuilder.setConfig(DockerPullConfigSerializer.toMesos(pullConfig))
aquamatthiasUnsubmitted
Done

Please add a test for:

val app = AppDefinitionWithDockerSecret
val roundtrip = app -> toProto -> fromProto
roundtrip == app
ichernetskyAuthorUnsubmitted
Not Done

Done.

359398 }
360399
361400 val imageBuilder = mesos.Protos.Image.newBuilder
362401 .setType(mesos.Protos.Image.Type.DOCKER)
363402 .setDocker(dockerBuilder)
364403 .setCached(!container.forcePullImage)
365404
366405 mesos.Protos.ContainerInfo.MesosInfo.newBuilder.setImage(imageBuilder).build
▲ Show 20 Lines • Show All 42 Lines • Show Last 20 Lines
View Options

src/main/scala/mesosphere/marathon/api/serialization/SecretSerializer.scala

  • This file was added.
1package mesosphere.marathon
2package api.serialization
3
4import org.apache.mesos.{ Protos => mesos }
5
6object SecretSerializer {
7 def toSecretReference(secret: String): mesos.Secret = {
8 val builder = mesos.Secret.newBuilder
9 builder.setType(mesos.Secret.Type.REFERENCE)
10 val referenceBuilder = mesos.Secret.Reference.newBuilder
11 referenceBuilder.setName(secret)
12 builder.setReference(referenceBuilder)
13 builder.build
14 }
15}
View Options

src/main/scala/mesosphere/marathon/api/v2/validation/AppValidation.scala

Show First 20 Lines • Show All 46 Lines • ▼ Show 20 Line(s)
4747 portMappings is portMappingsIndependentOfNetworks
4848 portMappings is every(portMappingIsCompatibleWithNetworks(networks))
4949 portMappings is every(portMappingNetworkNameValidator(networks.flatMap(_.name)))
5050 }
5151
5252 def dockerDockerContainerValidator(networks: Seq[Network]): Validator[Container] = {
5353 val validDockerEngineSpec: Validator[DockerContainer] = validator[DockerContainer] { docker =>
5454 docker.image is notEmpty
55 docker.pullConfig is isTrue("pullConfig is not supported with Docker containerizer")(_.isEmpty)
jdefUnsubmitted
Done

i don't see a validation test case for this. is it there and missed it? if not, please add one

5556 docker.portMappings is valid(optional(portMappingsValidator(networks)))
5657 }
5758 validator { (container: Container) =>
5859 container.docker is definedAnd(validDockerEngineSpec)
5960 }
6061 }
6162
62 val mesosDockerContainerValidator: Validator[Container] = {
63 def mesosDockerContainerValidator(secrets: Map[String, SecretDef]): Validator[Container] = {
64 val validPullConfigSpec: Validator[DockerPullConfig] =
65 isTrue("pullConfig.secret must refer to an existing secret")(
66 config => secrets.contains(config.secret))
6367 val validMesosEngineSpec: Validator[DockerContainer] = validator[DockerContainer] { docker =>
6468 docker.image is notEmpty
69 docker.pullConfig is optional(valid(validPullConfigSpec))
6570 }
66 validator{ (container: Container) =>
71 validator { (container: Container) =>
6772 container.docker is valid(definedAnd(validMesosEngineSpec))
6873 }
6974 }
7075
7176 val mesosAppcContainerValidator: Validator[Container] = {
7277 val prefix = "sha512-"
7378
7479 val validId: Validator[String] =
▲ Show 20 Lines • Show All 41 Lines • ▼ Show 20 Line(s)
116121 (container.docker, container.appc, container.`type`) match {
117122 case (Some(_), None, EngineType.Docker) => validate(container)(forDockerContainerizer)
118123 case (Some(_), None, EngineType.Mesos) => validate(container)(forMesosContainerizer)
119124 case _ => Success // canonical validation picks up where we leave off
120125 }
121126 }
122127 }
123128
124 def validContainer(enabledFeatures: Set[String], networks: Seq[Network]): Validator[Container] = {
129 def validContainer(enabledFeatures: Set[String], networks: Seq[Network], secrets: Map[String, SecretDef]): Validator[Container] = {
125130 def volumesValidator(container: Container): Validator[Seq[AppVolume]] =
126131 isTrue("Volume names must be unique") { (vols: Seq[AppVolume]) =>
127132 val names: Seq[String] = vols.flatMap(_.external.flatMap(_.name))
128133 names.distinct.size == names.size
129134 } and every(valid(validVolume(container, enabledFeatures)))
130135
131136 val validGeneralContainer: Validator[Container] = validator[Container] { container =>
132137 container.portMappings is optional(portMappingsValidator(networks))
133138 container.volumes is volumesValidator(container)
134139 }
135140
136141 val mesosContainerImageValidator = new Validator[Container] {
137142 override def apply(container: Container): Result = {
138143 (container.docker, container.appc, container.`type`) match {
139 case (Some(_), None, EngineType.Mesos) => validate(container)(mesosDockerContainerValidator)
144 case (Some(_), None, EngineType.Mesos) => validate(container)(mesosDockerContainerValidator(secrets))
140145 case (None, Some(_), EngineType.Mesos) => validate(container)(mesosAppcContainerValidator)
141146 case (None, None, EngineType.Mesos) => validate(container)(mesosImagelessContainerValidator)
142147 case _ => Failure(Set(RuleViolation(container, "mesos containers should specify, at most, a single image type", None)))
143148 }
144149 }
145150 }
146151
147152 forAll(
▲ Show 20 Lines • Show All 148 Lines • ▼ Show 20 Line(s)
296301 update.dependencies.map(_.map(PathId(_))) as "dependencies" is optional(every(valid))
297302 update.env is optional(envValidator(strictNameValidation = false, update.secrets.getOrElse(Map.empty), enabledFeatures))
298303 update.secrets is optional({ secrets: Map[String, SecretDef] =>
299304 secrets.nonEmpty
300305 } -> (featureEnabled(enabledFeatures, Features.SECRETS)))
301306 update.secrets is optional(featureEnabledImplies(enabledFeatures, Features.SECRETS)(every(secretEntryValidator)))
302307 update.fetch is optional(every(valid))
303308 update.portDefinitions is optional(portDefinitionsValidator)
304 update.container is optional(valid(validContainer(enabledFeatures, update.networks.getOrElse(Nil))))
309 update.container is optional(valid(validContainer(enabledFeatures, update.networks.getOrElse(Nil), update.secrets.getOrElse(Map.empty))))
305310 update.acceptedResourceRoles is valid(optional(ResourceRole.validAcceptedResourceRoles(update.residency.isDefined) and notEmpty))
306311 },
307312 isTrue("must not be root")(!_.id.fold(false)(PathId(_).isRoot)),
308313 isTrue("must not be an empty string")(_.cmd.forall { s => s.length() > 1 }),
309314 isTrue("portMappings are not allowed with host-networking") { update =>
310315 !(update.networks.exists(_.exists(_.mode == NetworkMode.Host)) && update.container.exists(_.portMappings.exists(_.nonEmpty)))
311316 },
312317 isTrue("portDefinitions are only allowed with host-networking") { update =>
▲ Show 20 Lines • Show All 66 Lines • ▼ Show 20 Line(s)
379384 def portIndices(app: App): Range = {
380385 // should be kept in sync with AppDefinition.portIndices
381386 app.container.withFilter(_.portMappings.nonEmpty)
382387 .flatMap(_.portMappings).orElse(app.portDefinitions).getOrElse(Nil).indices
383388 }
384389
385390 /** validate most canonical API fields */
386391 private def validBasicAppDefinition(enabledFeatures: Set[String]): Validator[App] = validator[App] { app =>
387 app.container is optional(valid(validContainer(enabledFeatures, app.networks)))
392 app.container is optional(valid(validContainer(enabledFeatures, app.networks, app.secrets)))
388393 app.portDefinitions is optional(portDefinitionsValidator)
389394 app is containsCmdArgsOrContainer
390395 app.healthChecks is every(portIndexIsValid(portIndices(app)))
391396 app must haveAtMostOneMesosHealthCheck
392397 app.fetch is every(valid)
393398 app.secrets is valid({ secrets: Map[String, SecretDef] =>
394399 secrets.nonEmpty
395400 } -> (featureEnabled(enabledFeatures, Features.SECRETS)))
▲ Show 20 Lines • Show All 126 Lines • Show Last 20 Lines
View Options

src/main/scala/mesosphere/marathon/api/v2/validation/PodsValidation.scala

11package mesosphere.marathon
22package api.v2.validation
33
44// scalastyle:off
55
66import com.wix.accord._
77import com.wix.accord.dsl._
88import mesosphere.marathon.api.v2.Validation
9import mesosphere.marathon.raml.{ ArgvCommand, Artifact, CommandHealthCheck, Endpoint, FixedPodScalingPolicy, HealthCheck, HttpHealthCheck, Image, ImageType, Lifecycle, Network, NetworkMode, Pod, PodContainer, PodScalingPolicy, Resources, ShellCommand, TcpHealthCheck, Volume, VolumeMount }
9import mesosphere.marathon.raml.{ ArgvCommand, Artifact, CommandHealthCheck, Endpoint, FixedPodScalingPolicy, HealthCheck, HttpHealthCheck, Image, ImageType, Lifecycle, Network, NetworkMode, Pod, PodContainer, PodScalingPolicy, Resources, SecretDef, ShellCommand, TcpHealthCheck, Volume, VolumeMount }
1010import mesosphere.marathon.state.PathId
1111import mesosphere.marathon.util.SemanticVersion
1212
1313// scalastyle:on
1414
1515/**
1616 * Defines implicit validation for pods
1717 */
▲ Show 20 Lines • Show All 91 Lines • ▼ Show 20 Line(s)
109109 endpoint.protocol is isTrue ("Duplicate protocols within the same endpoint are not allowed") { proto =>
110110 proto == proto.distinct
111111 }
112112 }
113113
114114 normalValidation and implied(networks.count(_.mode == NetworkMode.Container) > 1)(hostPortRequiresNetworkName)
115115 }
116116
117 val imageValidator = validator[Image] { image =>
118 image.id.length is between(1, 1024)
117 def imageValidator(secrets: Map[String, SecretDef]): Validator[Image] = new Validator[Image] {
118 override def apply(image: Image): Result = {
119 val dockerImageValidator: Validator[Image] = validator[Image] { image =>
jdefUnsubmitted
Done

nit: could simplify this (_.isEmpty) instead of { pullConfig => ...}

120 image.pullConfig is optional(
121 isTrue("pullConfig.secret must refer to an existing secret")(
122 config => secrets.contains(config.secret)))
119123 }
120124
125 val appcImageValidator: Validator[Image] = validator[Image] { image =>
126 image.pullConfig is isTrue("pullConfig is supported only with Docker images")(_.isEmpty)
127 }
jdefUnsubmitted
Not Done

Pretty sure that image.id validation is already handled by the generated RAML code:

val id = json.\("id").validate[String](play.api.libs.json.JsPath.read[String](maxLength[String](1024) keepAnd minLength[String](1)))

so .. no need to duplicate that validation here. the length check above (in the dockerImageValidator) code is redundant, and there's already a JIRA filed for removing redundant validation. we shouldn't add any more

128
129 image.kind match {
130 case ImageType.Docker => validate(image)(dockerImageValidator)
131 case ImageType.Appc => validate(image)(appcImageValidator)
132 }
133 }
134 }
jdefUnsubmitted
Done

this case _ is superfluous since ImageType is a sealed trait

135
121136 def volumeMountValidator(volumes: Seq[Volume]): Validator[VolumeMount] = validator[VolumeMount] { volumeMount => // linter:ignore:UnusedParameter
122137 volumeMount.name.length is between(1, 63)
123138 volumeMount.name should matchRegexFully(NamePattern)
124139 volumeMount.mountPath.length is between(1, 1024)
125140 volumeMount.name is isTrue("Referenced Volume in VolumeMount should exist") { name =>
126141 volumes.exists(_.name == name)
127142 }
128143 }
129144
130145 val artifactValidator = validator[Artifact] { artifact =>
131146 artifact.uri.length is between(1, 1024)
132147 artifact.destPath.map(_.length).getOrElse(1) is between(1, 1024)
133148 }
134149
135150 val lifeCycleValidator = validator[Lifecycle] { lc =>
136151 lc.killGracePeriodSeconds.getOrElse(0.0) should be > 0.0
137152 }
138153
139154 def containerValidator(pod: Pod, enabledFeatures: Set[String], mesosMasterVersion: SemanticVersion): Validator[PodContainer] =
140155 validator[PodContainer] { container =>
141156 container.resources is valid(resourceValidator)
142157 container.endpoints is every(endpointValidator(pod.networks))
143 container.image.getOrElse(Image(ImageType.Docker, "abc")) is valid(imageValidator)
158 container.image is optional(imageValidator(pod.secrets))
jdefUnsubmitted
Not Done

wow, we really left abc in there, eh?

144159 container.environment is envValidator(strictNameValidation = false, pod.secrets, enabledFeatures)
145160 container.healthCheck is optional(healthCheckValidator(container.endpoints, mesosMasterVersion))
146161 container.volumeMounts is every(volumeMountValidator(pod.volumes))
147162 container.artifacts is every(artifactValidator)
148163 }
149164
150165 def volumeValidator(containers: Seq[PodContainer]): Validator[Volume] = validator[Volume] { volume =>
151166 volume.host is optional(notEmpty)
▲ Show 20 Lines • Show All 65 Lines • Show Last 20 Lines
View Options

src/main/scala/mesosphere/marathon/raml/ContainerConversion.scala

Show First 20 Lines • Show All 49 Lines • ▼ Show 20 Line(s)
5050 }
5151
5252 implicit val containerWrites: Writes[state.Container, Container] = Writes { container =>
5353
5454 implicit val credentialWrites: Writes[state.Container.Credential, DockerCredentials] = Writes { credentials =>
5555 DockerCredentials(credentials.principal, credentials.secret)
5656 }
5757
58 implicit val pullConfigWrites: Writes[state.Container.DockerPullConfig, DockerPullConfig] = Writes {
59 case state.Container.DockerPullConfig(secret) => DockerPullConfig(secret)
60 }
61
5862 implicit val dockerDockerContainerWrites: Writes[state.Container.Docker, DockerContainer] = Writes { container =>
5963 DockerContainer(
6064 forcePullImage = container.forcePullImage,
6165 image = container.image,
6266 parameters = container.parameters.toRaml,
6367 privileged = container.privileged)
6468 }
6569
6670 implicit val mesosDockerContainerWrites: Writes[state.Container.MesosDocker, DockerContainer] = Writes { container =>
6771 DockerContainer(
6872 image = container.image,
6973 credential = container.credential.toRaml,
74 pullConfig = container.pullConfig.toRaml,
7075 forcePullImage = container.forcePullImage)
7176 }
7277
7378 implicit val mesosContainerWrites: Writes[state.Container.MesosAppC, AppCContainer] = Writes { container =>
7479 AppCContainer(container.image, container.id, container.labels, container.forcePullImage)
7580 }
7681
7782 def create(kind: EngineType, docker: Option[DockerContainer] = None, appc: Option[AppCContainer] = None): Container = {
7883 Container(kind, docker = docker, appc = appc, volumes = container.volumes.toRaml,
7984 portMappings = Option(container.portMappings.toRaml) // this might need to be None, but we can't check networking here
8085 )
8186 }
8287
8388 container match {
8489 case docker: state.Container.Docker => create(EngineType.Docker, docker = Some(docker.toRaml[DockerContainer]))
8590 case mesos: state.Container.MesosDocker => create(EngineType.Mesos, docker = Some(mesos.toRaml[DockerContainer]))
8691 case mesos: state.Container.MesosAppC => create(EngineType.Mesos, appc = Some(mesos.toRaml[AppCContainer]))
87 case mesos: state.Container.Mesos => create(EngineType.Mesos)
92 case _: state.Container.Mesos => create(EngineType.Mesos)
8893 }
8994 }
9095
96 implicit val pullConfigReads: Reads[DockerPullConfig, state.Container.DockerPullConfig] = Reads {
97 case DockerPullConfig(secret) => state.Container.DockerPullConfig(secret)
98 }
99
91100 implicit val appContainerRamlReader: Reads[Container, state.Container] = Reads { container =>
92101 val volumes = container.volumes.map(Raml.fromRaml(_))
93102 val portMappings: Seq[state.Container.PortMapping] = container.portMappings.getOrElse(Nil).map(Raml.fromRaml(_))
94103
95104 val result: state.Container = (container.`type`, container.docker, container.appc) match {
96105 case (EngineType.Docker, Some(docker), None) =>
97106 state.Container.Docker(
98107 volumes = volumes,
99108 image = docker.image,
100109 portMappings = portMappings, // assumed already normalized, see AppNormalization
101110 privileged = docker.privileged,
102111 parameters = docker.parameters.map(p => Parameter(p.key, p.value)),
103112 forcePullImage = docker.forcePullImage
104113 )
105114 case (EngineType.Mesos, Some(docker), None) =>
106115 state.Container.MesosDocker(
107116 volumes = volumes,
108117 image = docker.image,
109118 portMappings = portMappings, // assumed already normalized, see AppNormalization
110119 credential = docker.credential.map(c => state.Container.Credential(principal = c.principal, secret = c.secret)),
120 pullConfig = docker.pullConfig.map(_.fromRaml),
111121 forcePullImage = docker.forcePullImage
112122 )
113123 case (EngineType.Mesos, None, Some(appc)) =>
114124 state.Container.MesosAppC(
115125 volumes = volumes,
116126 image = appc.image,
117127 portMappings = portMappings,
118128 id = appc.id,
Show All 32 Lines
151161 DockerParameter(
152162 key = param.getKey,
153163 value = param.getValue
154164 )
155165 }
156166
157167 implicit val dockerProtoToRamlWriter: Writes[Protos.ExtendedContainerInfo.DockerInfo, DockerContainer] = Writes { docker =>
158168 DockerContainer(
159 credential = DockerContainer.DefaultCredential, // we don't store credentials in protobuf
169 credential = DockerContainer.DefaultCredential, // we don't store credentials in protobuf for Docker containerizer
170 pullConfig = DockerContainer.DefaultPullConfig, // we don't store a Docker config in protobuf for Docker containerizer
aquamatthiasUnsubmitted
Not Done

Didn't you add this to marathon.proto?

ichernetskyAuthorUnsubmitted
Not Done

It is a copy & pasted. I did add to the .proto file, but on the other hand, credential is there as well. Is the other one outdated?

jdefUnsubmitted
Done

"... for docker containerizer"

jdefUnsubmitted
Done

please amend the comment with something similar to the proposed clarification

160171 forcePullImage = docker.when(_.hasForcePullImage, _.getForcePullImage).getOrElse(DockerContainer.DefaultForcePullImage),
161172 image = docker.getImage,
162173 network = docker.when(_.hasOBSOLETENetwork, _.getOBSOLETENetwork.toRaml).orElse(DockerContainer.DefaultNetwork),
163174 parameters = docker.whenOrElse(_.getParametersCount > 0, _.getParametersList.map(_.toRaml)(collection.breakOut), DockerContainer.DefaultParameters),
164175 portMappings = Option.empty[Seq[ContainerPortMapping]].unless(
165176 docker.when(_.getOBSOLETENetwork != Mesos.ContainerInfo.DockerInfo.Network.HOST, _.getOBSOLETEPortMappingsList.map(_.toRaml)(collection.breakOut))),
166177 privileged = docker.when(_.hasPrivileged, _.getPrivileged).getOrElse(DockerContainer.DefaultPrivileged)
167178 )
168179 }
169180
170181 implicit val dockerCredentialsProtoRamlWriter: Writes[Mesos.Credential, DockerCredentials] = Writes { cred =>
171182 DockerCredentials(
172183 principal = cred.getPrincipal,
173184 secret = cred.when(_.hasSecret, _.getSecret).orElse(DockerCredentials.DefaultSecret)
174185 )
175186 }
176187
188 implicit val dockerPullConfigProtoRamlWriter: Writes[Protos.ExtendedContainerInfo.DockerInfo.ImagePullConfig, DockerPullConfig] = Writes { pullConfig =>
189 pullConfig.when(_.getType == Protos.ExtendedContainerInfo.DockerInfo.ImagePullConfig.Type.SECRET, _ => {
190 pullConfig.when(_.hasSecret, _.getSecret).flatMap { secret =>
191 secret.when(_.hasType, _.getType).flatMap {
192 case Mesos.Secret.Type.REFERENCE =>
jdefUnsubmitted
Done

ditto, don't swallow unsupported type here: better to throw an exception instead

193 secret.when(_.hasReference, _.getReference.getName).map(DockerPullConfig(_))
194 case _ => None
195 }
196 }
197 }).flatten match {
198 case Some(deserializedPullConfig) => deserializedPullConfig
199 case _ =>
200 throw SerializationFailedException(s"Failed to deserialize a docker pull config: $pullConfig")
jdefUnsubmitted
Done

ditto, new is not required here

201 }
202 }
203
177204 implicit val mesosDockerProtoToRamlWriter: Writes[Protos.ExtendedContainerInfo.MesosDockerInfo, DockerContainer] = Writes { docker =>
178205 DockerContainer(
179 credential = docker.when(_.hasCredential, _.getCredential.toRaml).orElse(DockerContainer.DefaultCredential),
206 credential = docker.when(_.hasDeprecatedCredential, _.getDeprecatedCredential.toRaml).orElse(DockerContainer.DefaultCredential),
207 pullConfig = docker.when(_.hasPullConfig, _.getPullConfig.toRaml).orElse(DockerContainer.DefaultPullConfig),
180208 forcePullImage = docker.when(_.hasForcePullImage, _.getForcePullImage).getOrElse(DockerContainer.DefaultForcePullImage),
181209 image = docker.getImage
182210 // was never stored for mesos containers:
183211 // - network
184212 // - parameters
185213 // - portMappings
186214 // - privileged
187215 )
Show All 21 Lines
View Options

src/main/scala/mesosphere/marathon/state/Container.scala

Show First 20 Lines • Show All 88 Lines • ▼ Show 20 Line(s)
8989
9090 val HostPortDefault = AppDefinition.RandomPortValue // HostPortDefault only applies when in BRIDGE mode
9191 }
9292
9393 case class Credential(
9494 principal: String,
9595 secret: Option[String] = None)
9696
97 case class DockerPullConfig(secret: String)
jdefUnsubmitted
Not Done

ditto on naming

98
9799 case class MesosDocker(
98100 volumes: Seq[Volume] = Seq.empty,
99101 image: String = "",
100102 override val portMappings: Seq[PortMapping] = Nil,
101103 credential: Option[Credential] = None,
104 pullConfig: Option[DockerPullConfig] = None,
102105 forcePullImage: Boolean = false) extends Container {
103106
104107 override def copyWith(portMappings: Seq[PortMapping] = portMappings, volumes: Seq[Volume] = volumes) =
105108 copy(portMappings = portMappings, volumes = volumes)
106109 }
107110
108111 object MesosDocker {
109112 val validMesosDockerContainer = validator[MesosDocker] { docker =>
▲ Show 20 Lines • Show All 53 Lines • Show Last 20 Lines
View Options

src/test/scala/mesosphere/marathon/api/v2/AppsResourceTest.scala

11package mesosphere.marathon
22package api.v2
33
44import java.util
55import javax.ws.rs.core.Response
66
77import akka.Done
88import mesosphere.AkkaUnitTest
99import mesosphere.marathon.api._
1010import mesosphere.marathon.core.appinfo.AppInfo.Embed
1111import mesosphere.marathon.core.appinfo._
1212import mesosphere.marathon.core.base.ConstantClock
1313import mesosphere.marathon.core.deployment.DeploymentPlan
1414import mesosphere.marathon.core.group.GroupManager
1515import mesosphere.marathon.core.plugin.PluginManager
1616import mesosphere.marathon.core.pod.ContainerNetwork
17import mesosphere.marathon.raml.{ App, AppUpdate, ContainerPortMapping, DockerContainer, DockerNetwork, EngineType, EnvVarValueOrSecret, IpAddress, IpDiscovery, IpDiscoveryPort, Network, NetworkConversionMessages, NetworkMode, Raml, SecretDef }
17import mesosphere.marathon.raml.{ Container => RamlContainer }
18import mesosphere.marathon.raml.{ App, AppUpdate, ContainerPortMapping, DockerContainer, DockerNetwork, DockerPullConfig, EngineType, EnvVarValueOrSecret, IpAddress, IpDiscovery, IpDiscoveryPort, Network, NetworkConversionMessages, NetworkMode, Raml, SecretDef }
1819import mesosphere.marathon.state.PathId._
1920import mesosphere.marathon.state._
2021import mesosphere.marathon.storage.repository.GroupRepository
2122import mesosphere.marathon.test.GroupCreation
2223import org.mockito.Matchers
2324import play.api.libs.json._
2425
2526import scala.collection.immutable
▲ Show 20 Lines • Show All 131 Lines • ▼ Show 20 Line(s)
157158 if (!result.isSuccess) {
158159 result.failed.foreach {
159160 case v: ValidationFailedException =>
160161 assert(result.isSuccess, s"JSON body = ${new String(body)} :: violations = ${v.failure.violations}")
161162 case th =>
162163 throw th
163164 }
164165 }
166 }
167
168 "Create a new app with w/ Mesos containerizer and a Docker config.json" in new Fixture(configArgs = Seq("--enable_features", "secrets")) {
jdefUnsubmitted
Not Done

happy to see some tests here. i'd like a more thorough exercise of the validation rules to appear in AppValidationTest

169 Given("An app with a Docker config.json")
170 val container = RamlContainer(
171 `type` = EngineType.Mesos,
172 docker = Option(DockerContainer(
173 image = "private/image",
174 pullConfig = Option(DockerPullConfig("pullConfigSecret")))))
175 val app = App(
176 id = "/app", cmd = Some("cmd"), container = Option(container),
177 secrets = Map("pullConfigSecret" -> SecretDef("/config")))
178 val (body, plan) = prepareApp(app, groupManager)
179
180 When("The create request is made")
181 clock += 5.seconds
182 val response = appsResource.create(body, force = false, auth.request)
183 val result = Try(prepareApp(app, groupManager))
184
185 Then("It is successful")
186 assert(response.getStatus == 201, s"body=${new String(body)}, response=${response.getEntity.asInstanceOf[String]}")
187 }
188
189 "Creating a new app with w/ Docker containerizer and a Docker config.json should fail" in new Fixture(configArgs = Seq("--enable_features", "secrets")) {
190 Given("An app with a Docker config.json")
191 val container = RamlContainer(
192 `type` = EngineType.Docker,
193 docker = Option(DockerContainer(
194 image = "private/image",
195 pullConfig = Option(DockerPullConfig("pullConfigSecret")))))
196 val app = App(
197 id = "/app", cmd = Some("cmd"), container = Option(container),
198 secrets = Map("pullConfigSecret" -> SecretDef("/config")))
199 val (body, plan) = prepareApp(app, groupManager)
200
201 When("The create request is made")
202 clock += 5.seconds
203 val response = appsResource.create(body, force = false, auth.request)
204 val result = Try(prepareApp(app, groupManager))
205
206 Then("It fails")
207 assert(response.getStatus == 422, s"body=${new String(body)}, response=${response.getEntity.asInstanceOf[String]}")
208 response.getEntity.toString should include("pullConfig is not supported with Docker containerizer")
209 }
210
211 "Creating a new app with non-existing Docker config.json secret should fail" in new Fixture(configArgs = Seq("--enable_features", "secrets")) {
212 Given("An app with a Docker config.json")
213 val container = RamlContainer(
214 `type` = EngineType.Mesos,
215 docker = Option(DockerContainer(
216 image = "private/image",
217 pullConfig = Option(DockerPullConfig("pullConfigSecret")))))
218 val app = App(
219 id = "/app", cmd = Some("cmd"), container = Option(container))
220 val (body, plan) = prepareApp(app, groupManager)
221
222 When("The create request is made")
223 clock += 5.seconds
224 val response = appsResource.create(body, force = false, auth.request)
225 val result = Try(prepareApp(app, groupManager))
226
227 Then("It fails")
228 assert(response.getStatus == 422, s"body=${new String(body)}, response=${response.getEntity.asInstanceOf[String]}")
229 response.getEntity.toString should include("pullConfig.secret must refer to an existing secret")
165230 }
166231
167232 "Do partial update with patch methods" in new Fixture {
168233 Given("An app")
169234 val id = "/app"
170235 val app = App(
171236 id = id,
172237 cmd = Some("cmd"),
▲ Show 20 Lines • Show All 1201 Lines • Show Last 20 Lines
View Options

src/test/scala/mesosphere/marathon/api/v2/PodsResourceTest.scala

Show First 20 Lines • Show All 231 Lines • ▼ Show 20 Line(s)
232232 withClue(s"response body: ${response.getEntity}") {
233233 response.getStatus should be(HttpServletResponse.SC_NOT_FOUND)
234234 val body = Option(response.getEntity.asInstanceOf[String])
235235 body should not be None
236236 body.foreach(_ should include("mypod does not exist"))
237237 }
238238 }
239239
240 "Create a new pod with w/ Docker image and config.json" in {
jdefUnsubmitted
Not Done

happy to see some tests here. i'd like a more thorough exercise of the validation rules to appear in PodsValidationTest

241 implicit val podSystem = mock[PodManager]
242 val f = Fixture(configArgs = Seq("--enable_features", "secrets"))
243
244 podSystem.create(any, eq(false)).returns(Future.successful(DeploymentPlan.empty))
245
246 val podJson =
247 """
248 |{
249 | "id": "/pod",
250 | "containers": [{
251 | "name": "container0",
252 | "resources": {
253 | "cpus": 0.1,
254 | "mem": 32
255 | },
256 | "image": {
257 | "kind": "DOCKER",
258 | "id": "private/image",
259 | "pullConfig": {
260 | "secret": "pullConfigSecret"
261 | }
262 | },
263 | "exec": {
264 | "command": {
265 | "shell": "sleep 1"
266 | }
267 | }
268 | }],
269 | "secrets": {
270 | "pullConfigSecret": {
271 | "source": "/config"
272 | }
273 | }
274 |}
275 """.stripMargin
276
277 val response = f.podsResource.create(podJson.getBytes(), force = false, f.auth.request)
278
279 withClue(s"response body: ${response.getEntity}") {
280 response.getStatus should be(HttpServletResponse.SC_CREATED)
281
282 val parsedResponse = Option(response.getEntity.asInstanceOf[String]).map(Json.parse)
283 parsedResponse should be (defined)
284 val maybePod = parsedResponse.map(_.as[Pod])
285 maybePod should be (defined) // validate that we DID get back a pod definition
286 val pod = maybePod.get
287 pod.containers.headOption should be (defined)
288 val container = pod.containers.head
289 container.image should be (defined)
290 val image = container.image.get
291 image.pullConfig should be (defined)
292 val pullConfig = image.pullConfig.get
293 pullConfig.secret should be ("pullConfigSecret")
294
295 response.getMetadata.containsKey(RestResource.DeploymentHeader) should be(true)
296 }
297 }
298
299 "Creating a new pod with w/ AppC image and config.json should fail" in {
300 implicit val podSystem = mock[PodManager]
301 val f = Fixture(configArgs = Seq("--enable_features", "secrets"))
302
303 podSystem.create(any, eq(false)).returns(Future.successful(DeploymentPlan.empty))
304
305 val podJson =
306 """
307 |{
308 | "id": "/pod",
309 | "containers": [{
310 | "name": "container0",
311 | "resources": {
312 | "cpus": 0.1,
313 | "mem": 32
314 | },
315 | "image": {
316 | "kind": "APPC",
317 | "id": "private/image",
318 | "pullConfig": {
319 | "secret": "pullConfigSecret"
320 | }
321 | },
322 | "exec": {
323 | "command": {
324 | "shell": "sleep 1"
325 | }
326 | }
327 | }],
328 | "secrets": {
329 | "pullConfigSecret": {
330 | "source": "/config"
331 | }
332 | }
333 |}
334 """.stripMargin
335
336 val response = f.podsResource.create(podJson.getBytes(), force = false, f.auth.request)
337
338 withClue(s"response body: ${response.getEntity}") {
339 response.getStatus should be(422)
340 response.getEntity.toString should include("pullConfig is supported only with Docker images")
341 }
342 }
343
344 "Creating a new pod with w/ Docker image and non-existing secret should fail" in {
345 implicit val podSystem = mock[PodManager]
346 val f = Fixture(configArgs = Seq("--enable_features", "secrets"))
347
348 podSystem.create(any, eq(false)).returns(Future.successful(DeploymentPlan.empty))
349
350 val podJson =
351 """
352 |{
353 | "id": "/pod",
354 | "containers": [{
355 | "name": "container0",
356 | "resources": {
357 | "cpus": 0.1,
358 | "mem": 32
359 | },
360 | "image": {
361 | "kind": "Docker",
362 | "id": "private/image",
363 | "pullConfig": {
364 | "secret": "pullConfigSecret"
365 | }
366 | },
367 | "exec": {
368 | "command": {
369 | "shell": "sleep 1"
370 | }
371 | }
372 | }],
373 | "secrets": {
374 | "pullConfigSecretA": {
375 | "source": "/config"
376 | }
377 | }
378 |}
379 """.stripMargin
380
381 val response = f.podsResource.create(podJson.getBytes(), force = false, f.auth.request)
382
383 withClue(s"response body: ${response.getEntity}") {
384 response.getStatus should be(422)
385 response.getEntity.toString should include("pullConfig.secret must refer to an existing secret")
386 }
387 }
388
240389 "support versions" when {
241390 implicit val ctx = ExecutionContexts.global
242391
243392 "there are no versions" when {
244393 "list no versions" in {
245394 val groupManager = mock[GroupManager]
246395 groupManager.pod(any).returns(None)
247396 implicit val podManager = PodManagerImpl(groupManager)
▲ Show 20 Lines • Show All 244 Lines • Show Last 20 Lines
View Options

src/test/scala/mesosphere/marathon/api/v2/validation/AppValidationTest.scala

11package mesosphere.marathon
22package api.v2.validation
33
44import com.wix.accord.scalatest.ResultMatchers
55import mesosphere.{ UnitTest, ValidationTestLike }
66import mesosphere.marathon.raml._
77import mesosphere.UnitTest
88
99class AppValidationTest extends UnitTest with ResultMatchers with ValidationTestLike {
1010
1111 import Normalization._
1212
13 "Docker image pull config validation" when {
14 implicit val basicValidator = AppValidation.validateCanonicalAppAPI(Set("secrets"))
15
16 "pull config when the Mesos containerizer is used and the corresponding secret is provided" should {
17 "be accepted" in {
18 val app = App(
19 id = "/foo",
20 cmd = Some("bar"),
21 container = Some(Container(
22 `type` = EngineType.Mesos,
23 docker = Some(DockerContainer(
24 image = "xyz", pullConfig = Some(DockerPullConfig("aSecret")))))),
25 secrets = Map("aSecret" -> SecretDef("/secret")))
26
27 basicValidator(app) shouldBe (aSuccess)
28 }
29 }
30
31 "pull config when the Mesos containerizer is used and the corresponding secret is not provided" should {
32 "fail" in {
33 val app = App(
34 id = "/foo",
35 cmd = Some("bar"),
36 container = Some(Container(
37 `type` = EngineType.Mesos,
38 docker = Some(DockerContainer(
39 image = "xyz", pullConfig = Some(DockerPullConfig("aSecret")))))))
40
41 basicValidator(app).normalize should failWith(
42 "/container/docker/pullConfig" ->
43 "pullConfig.secret must refer to an existing secret")
44 }
45 }
46
47 "pull config when the Docker containerizer is used and the corresponding secret is provided" should {
48 "fail" in {
49 val app = App(
50 id = "/foo",
51 cmd = Some("bar"),
52 container = Some(Container(
53 `type` = EngineType.Docker,
54 docker = Some(DockerContainer(
55 image = "xyz", pullConfig = Some(DockerPullConfig("aSecret")))))),
56 secrets = Map("aSecret" -> SecretDef("/secret")))
57
58 basicValidator(app).normalize should failWith(
59 "/container/docker" ->
60 "pullConfig is not supported with Docker containerizer")
61 }
62 }
63 }
64
1365 "network validation" when {
1466 implicit val basicValidator = AppValidation.validateCanonicalAppAPI(Set.empty)
1567
1668 def networkedApp(portMappings: Seq[ContainerPortMapping], networks: Seq[Network], docker: Boolean = false) = {
1769 App(
1870 id = "/foo",
1971 cmd = Some("bar"),
2072 networks = networks,
▲ Show 20 Lines • Show All 148 Lines • Show Last 20 Lines
View Options

src/test/scala/mesosphere/marathon/raml/ContainerConversionTest.scala

Show All 40 Lines
4141 mc.portMappings should be(Seq(corePortMapping))
4242 mc.volumes should be(Seq(coreHostVolume))
4343 }
4444 }
4545 }
4646
4747 "A Mesos Docker container is converted" when {
4848 "a mesos-docker container" should {
49 val container = state.Container.MesosDocker(Seq(coreHostVolume), "test", Seq(corePortMapping), Some(credentials))
49 val container = state.Container.MesosDocker(Seq(coreHostVolume), "test", Seq(corePortMapping),
50 Some(credentials), Some(dockerPullConfig))
5051 val raml = container.toRaml[Container]
5152
5253 behave like convertToProtobufThenToRAML(container, raml)
5354
5455 "convert to a RAML container" in {
5556 raml.`type` should be(EngineType.Mesos)
5657 raml.appc should be(empty)
5758 raml.volumes should be(Seq(ramlHostVolume))
5859 raml.portMappings should contain(Seq(ramlPortMapping))
5960 raml.docker should be(defined)
6061 raml.docker.get.image should be("test")
6162 raml.docker.get.credential should be(defined)
6263 raml.docker.get.credential.get.principal should be(credentials.principal)
6364 raml.docker.get.credential.get.secret should be(credentials.secret)
65 raml.docker.get.pullConfig should be(defined)
66 raml.docker.get.pullConfig.get shouldBe a[DockerPullConfig]
67 raml.docker.get.pullConfig.get shouldBe DockerPullConfig(dockerPullConfig.secret)
6468 }
6569 }
6670 "a mesos-docker container w/o port mappings" should {
67 val container = state.Container.MesosDocker(Seq(coreHostVolume), "test", portMappings = Seq.empty, Some(credentials))
71 val container = state.Container.MesosDocker(Seq(coreHostVolume), "test", portMappings = Seq.empty,
72 Some(credentials), Some(dockerPullConfig))
6873 val raml = container.toRaml[Container]
6974 behave like convertToProtobufThenToRAML(container, raml)
7075 }
7176 "a RAML container" should {
7277 "convert to a mesos-docker container" in {
7378 val container = Container(EngineType.Mesos, portMappings = Option(Seq(ramlPortMapping)), docker = Some(DockerContainer(
74 image = "foo", credential = Some(DockerCredentials(credentials.principal, credentials.secret)))),
75 volumes = Seq(ramlHostVolume))
79 image = "foo", credential = Some(DockerCredentials(credentials.principal, credentials.secret)),
80 pullConfig = Some(DockerPullConfig(dockerPullConfig.secret)))), volumes = Seq(ramlHostVolume))
7681 val mc = Some(container.fromRaml).collect {
7782 case c: state.Container.MesosDocker => c
7883 }.getOrElse(fail("expected Container.MesosDocker"))
7984 mc.portMappings should be(Seq(corePortMapping))
8085 mc.volumes should be(Seq(coreHostVolume))
8186 mc.image should be("foo")
82 mc.credential should be(Some(credentials))
87 mc.credential shouldBe Some(credentials)
88 mc.pullConfig shouldBe Some(dockerPullConfig)
8389 mc.forcePullImage should be(container.docker.head.forcePullImage)
8490 }
8591 }
8692 }
8793
8894 "A Mesos AppC container is created correctly" when {
8995 "a mesos-appc container" should {
9096 val container = state.Container.MesosAppC(Seq(coreHostVolume), "test", Seq(corePortMapping), Some("id"))
▲ Show 20 Lines • Show All 115 Lines • ▼ Show 20 Line(s)
206212
207213 "convert to a RAML container" in {
208214 raml.`type` should be(EngineType.Docker)
209215 raml.appc should be(empty)
210216 raml.volumes should be(Seq(ramlHostVolume))
211217 raml.docker should be(defined)
212218 raml.docker.get.image should be("test")
213219 raml.docker.get.credential should be(empty)
220 raml.docker.get.pullConfig shouldBe empty
214221 raml.docker.get.network should be(empty)
215222 raml.portMappings should contain(Seq(ramlPortMapping))
216223 }
217224 }
218225 "a docker-docker container w/o port mappings" should {
219226 val container = state.Container.Docker(Seq(coreHostVolume), "test")
220227 val raml = container.toRaml[Container]
221228 behave like convertToProtobufThenToRAML(container, raml)
Show All 11 Lines
233240 mc.forcePullImage should be(container.docker.head.forcePullImage)
234241 mc.parameters should be(Seq(state.Parameter("qws", "erf")))
235242 mc.privileged should be(container.docker.head.privileged)
236243 }
237244 }
238245 }
239246
240247 private lazy val credentials = state.Container.Credential("principal", Some("secret"))
248 private lazy val dockerPullConfig = state.Container.DockerPullConfig("aConfigSecret")
241249 private lazy val ramlPortMapping = ContainerPortMapping(
242250 containerPort = 80,
243251 hostPort = Some(90),
244252 servicePort = 100,
245253 name = Some("pok"),
246254 labels = Map("wer" -> "rty")
247255 )
248256 private lazy val corePortMapping = state.Container.PortMapping(
Show All 9 Lines
View Options

src/test/scala/mesosphere/mesos/TaskBuilderTest.scala

Show First 20 Lines • Show All 623 Lines • ▼ Show 20 Line(s)
624624
625625 "build creates task for MESOS Docker container" in {
626626 val offer = MarathonTestHelper.makeBasicOfferWithRole(
627627 cpus = 1.0, mem = 128.0, disk = 1000.0, beginPort = 31000, endPort = 31010, role = ResourceRole.Unreserved
628628 )
629629 .addResources(RangesResource(Resource.PORTS, Seq(protos.Range(33000, 34000)), "marathon"))
630630 .build
631631
632 val dockerPullConfigSecret = "aConfigSecret"
633 val dockerPullConfig = Container.DockerPullConfig(dockerPullConfigSecret)
632634 val task: Option[(MesosProtos.TaskInfo, _)] = buildIfMatches(
633635 offer, AppDefinition(
634636 id = "/testApp".toPath,
635637 resources = Resources(cpus = 1.0, mem = 64.0, disk = 1.0),
636638 executor = "//cmd",
637639 container = Some(Container.MesosDocker(
638640 image = "busybox",
639641 credential = Some(Container.Credential(
640642 principal = "aPrincipal",
641643 secret = Some("aSecret")
642 ))
643644 )),
645 pullConfig = Some(dockerPullConfig)
aquamatthiasUnsubmitted
Not Done

Please add a test for secret ref

646 )),
644647 portDefinitions = Seq.empty,
645648 networks = Seq(ContainerNetwork("vnet"))
646649 )
647650 )
648651 assert(task.isDefined, "expected task to match offer")
649652 val (taskInfo: TaskInfo, _) = task.get
650653 taskInfo.hasContainer should be (true)
651654 taskInfo.getContainer.getType should be (MesosProtos.ContainerInfo.Type.MESOS)
652655 taskInfo.getContainer.hasMesos should be (true)
653656 taskInfo.getContainer.getMesos.hasImage should be (true)
654657 taskInfo.getContainer.getMesos.getImage.getType should be (MesosProtos.Image.Type.DOCKER)
655658 taskInfo.getContainer.getMesos.getImage.hasDocker should be (true)
656659 taskInfo.getContainer.getMesos.getImage.getDocker.hasCredential should be (true)
657660 taskInfo.getContainer.getMesos.getImage.getDocker.getCredential.getPrincipal should be ("aPrincipal")
658661 taskInfo.getContainer.getMesos.getImage.getDocker.getCredential.hasSecret should be (true)
659662 taskInfo.getContainer.getMesos.getImage.getDocker.getCredential.getSecret should be ("aSecret")
663 taskInfo.getContainer.getMesos.getImage.getDocker.hasConfig shouldBe true
664 taskInfo.getContainer.getMesos.getImage.getDocker.getConfig.getReference.getName shouldBe dockerPullConfigSecret
660665 }
661666
662667 "build creates task for MESOS AppC container" in {
663668 val offer = MarathonTestHelper.makeBasicOfferWithRole(
664669 cpus = 1.0, mem = 128.0, disk = 1000.0, beginPort = 31000, endPort = 31010, role = ResourceRole.Unreserved
665670 )
666671 .addResources(RangesResource(Resource.PORTS, Seq(protos.Range(33000, 34000)), "marathon"))
667672 .build
▲ Show 20 Lines • Show All 1278 Lines • Show Last 20 Lines
View Options

tests/system/common.py

Show All 12 Lines
1313import pytest
1414import shakedown
1515
1616
1717marathon_1_3 = pytest.mark.skipif('marthon_version_less_than("1.3")')
1818marathon_1_4 = pytest.mark.skipif('marthon_version_less_than("1.4")')
1919marathon_1_5 = pytest.mark.skipif('marthon_version_less_than("1.5")')
2020
21def escape_cli_arg(arg):
22 acc = []
23 for char in arg:
24 if char in ('"', '\\'):
25 acc.append('\\')
26 acc.append(char)
27 return ''.join(acc)
28
2129
2230def app(id=1, instances=1):
2331 app_json = {
2432 "id": "",
2533 "instances": 1,
2634 "cmd": "sleep 100000000",
2735 "cpus": 0.01,
2836 "mem": 1,
▲ Show 20 Lines • Show All 427 Lines • ▼ Show 20 Line(s)
456464 "fetch": [
457465 {
458466 "uri": "file:///home/core/{}".format(docker_credentials_filename)
459467 }
460468 ]
461469 }
462470
463471
464def private_mesos_container_app(principal, secret):
472def private_mesos_container_app(secret_name, app_id=None):
473 """ Returns an application definition that uses Mesos containerizer and
474 expects a valid Docker config.json referenced by the given `secret_name`.
475
476 :param secret_name: secret name which value is a Docker config.json
477 :param app_id: optional application ID, if not given, a random UUID is used
478 :return: application definition represented using Python data structures
479 """
480 if app_id is None:
481 app_id = '/{}'.format(uuid.uuid4().hex)
482
kensipeUnsubmitted
Done

I don't know what "private_mesos_app" means... can we rename this to something meaningful? or add docs that explains why it should be used?

zen-dogUnsubmitted
Done

App id should have a random part (see using of uuid in other tests). Prevents collisions and makes debugging easier.

kensipeUnsubmitted
Not Done

your comment doesn't address the comment or concern... the issue isn't app_id... it is the meaning of private_mesos_container and documentation

zen-dogUnsubmitted
Not Done

It was meant as an addition to your comment ;)

465483 return {
466 "id": "/private-mesos-app",
484 "id": app_id,
467485 "instances": 1,
468486 "cpus": 1,
469487 "mem": 128,
470488 "container": {
471 "type": 'MESOS',
472 "docker": {
473 "image": "mesosphere/simple-docker-ee:latest",
474 "credential": {
475 "principal": principal,
476 "secret": secret
489 "type": 'MESOS',
490 "docker": {
491 "image": "mesosphere/simple-docker-ee:latest",
492 "config": {
493 "secret": "pullConfigSecret"
494 }
495 }
496 },
497 "secrets": {
498 "pullConfigSecret": {
499 "source": secret_name
500 }
501 }
502 }
503
504
505def private_docker_pod(secret_name, pod_id=None):
506 """ Returns a pod definition that uses a Docker image and
507 expects a valid Docker config.json referenced by the given `secret_name`.
508
509 :param secret_name: secret name which value is a Docker config.json
510 :param pod_id: optional pod ID, if not given, a random UUID is used
511 :return: pod definition represented using Python data structures
512 """
513 if pod_id is None:
514 pod_id = '/{}'.format(uuid.uuid4().hex)
515
516 return {
517 "id": pod_id,
518 "scaling": {
519 "kind": "fixed",
520 "instances": 1
521 },
522 "containers": [{
523 "name": "simple-docker",
524 "resources": {
525 "cpus": 1,
526 "mem": 128
527 },
528 "image": {
529 "kind": "DOCKER",
530 "id": "mesosphere/simple-docker-ee:latest",
531 "config": {
532 "secret": "pullConfigSecret"
477533 }
478534 }
535 }],
536 "secrets": {
537 "pullConfigSecret": {
538 "source": secret_name
539 }
479540 }
480541 }
481542
482543
483544def pinger_localhost_app(id='pinger', port=7777):
484545 """ pinger app requires, the pinger.py app in fixure_dir and the master
485546 http service started at port 7777
486547
▲ Show 20 Lines • Show All 288 Lines • ▼ Show 20 Line(s)
775836def is_enterprise_cli_package_installed():
776837 """ Returns `True` if `dcos-enterprise-cli` package is installed.
777838 """
778839 stdout, stderr, return_code = run_dcos_command('package list --json')
779840 result_json = json.loads(stdout)
780841 return any(cmd['name'] == 'dcos-enterprise-cli' for cmd in result_json)
781842
782843
844def create_docker_pull_config_json(username, password):
845 """ Create a Docker config.json represented using Python data structures.
846
847 :param username: username for a private Docker registry
848 :param password: password for a private Docker registry
849 :return: Docker config.json
850 """
851 print('Creating a config.json content for dockerhub username {}'.format(username))
852
853 import base64
854 auth_hash = base64.b64encode('{}:{}'.format(username, password).encode()).decode()
855
856 return {
857 "auths": {
858 "https://index.docker.io/v1/": {
859 "auth": auth_hash
860 }
861 }
862 }
863
864
783865def create_docker_credentials_file(username, password, file_name='docker.tar.gz'):
kensipeUnsubmitted
Not Done

this is in shakedown... all of this can be replaced with shakedown. distribute_docker_credentials_to_private_agents()
https://github.com/dcos/shakedown/blob/master/shakedown/dcos/docker.py#L65

zen-dogUnsubmitted
Not Done

This is not the same - there is no need to distribute the file, since it's passed as part of the app definition. However we should extract creation part into a separate function and maybe move it to shakedown later.

ichernetskyAuthorUnsubmitted
Not Done

Aleksey is correct.

784866 """ Create a docker credentials file. Docker username and password are used to create
785867 a `{file_name}` with `.docker/config.json` containing the credentials.
786868
787869 :param file_name: credentials file name `docker.tar.gz` by default
788870 :type command: str
789871 """
790872
791873 print('Creating a tarball {} with json credentials for dockerhub username {}'.format(file_name, username))
792874 config_json_filename = 'config.json'
793875
794 import base64
876 config_json = create_docker_pull_config_json(username, password)
zen-dogUnsubmitted
Done

It seems like next 10 lines can be replaced by create_docker_pull_config_json() call?

ichernetskyAuthorUnsubmitted
Not Done

Yep, forgot to do it.

795 auth_hash = base64.b64encode('{}:{}'.format(username, password).encode()).decode()
796
797 config_json = {
798 "auths": {
799 "https://index.docker.io/v1/": {
800 "auth": auth_hash
801 }
802 }
803 }
804877
805878 # Write config.json to file
806879 with open(config_json_filename, 'w') as f:
807880 json.dump(config_json, f, indent=4)
808881
809882 try:
810883 # Create a docker.tar.gz
811884 import tarfile
Show All 25 Lines
837910 copy_file_to_agent(agent, file_name)
838911 except Exception as e:
839912 print('Failed to upload {} to agent: {}'.format(file_name, agent))
840913 raise e
841914 finally:
842915 os.remove(file_name)
843916
844917
918def create_secret(secret_name, secret_value):
919 """ Create a secret with a given name and a value.
920 This method uses `dcos security secrets` command and assumes that `dcos-enterprise-cli`
921 package is installed.
922
923 :param secret_name: secret name
924 :type secret_name: str
925 :param secret_value: secret_value
926 :type secret_value: str
927 """
928 escaped_secret_value = escape_cli_arg(secret_value)
929 stdout, stderr, return_code = run_dcos_command(
930 'security secrets create --value="{}" {}'.format(escaped_secret_value, secret_name))
931 assert return_code == 0, "Failed to create a secret: {}".format(secret_name)
932
933
845934def has_secret(secret_name):
846935 """ Returns `True` if the secret with given name exists in the vault.
847936 This method uses `dcos security secrets` command and assumes that `dcos-enterprise-cli`
848937 package is installed.
849938
850939 :param secret_name: secret name
851940 :type secret_name: str
852941 """
Show All 12 Lines
865954 :param secret_name: secret name
866955 :type secret_name: str
867956 """
868957 print('Removing existing secret {}'.format(secret_name))
869958 stdout, stderr, return_code = run_dcos_command('security secrets delete {}'.format(secret_name))
870959 assert return_code == 0, "Failed to remove existing secret"
871960
872961
873def create_secret(secret_name, service_account, strict=False, private_key_filename='private-key.pem'):
962def create_sa_secret(secret_name, service_account, strict=False, private_key_filename='private-key.pem'):
874963 """ Create a secret with a given private key file for passed service account in the vault. Both
875964 (service account and secret) should share the same key pair. `{strict}` parameter should be
876965 `True` when creating a secret in a `strict` secure cluster. Private key file will be removed
877966 after secret is successfully created.
878967 This method uses `dcos security org` command and assumes that `dcos-enterprise-cli`
879968 package is installed.
880969
881970 :param secret_name: secret name
▲ Show 20 Lines • Show All 132 Lines • Show Last 20 Lines
View Options

tests/system/marathon_common_tests.py

Show First 20 Lines • Show All 973 Lines • ▼ Show 20 Line(s)
974974 client = marathon.create_client()
975975 app_def = common.private_docker_container_app()
976976 client.add_app(app_def)
977977 shakedown.deployment_wait()
978978
979979 common.assert_app_tasks_running(client, app_def)
980980
981981
982@pytest.mark.skip(reason="Not yet implemented in mesos")
982@pytest.mark.skipif("docker_env_set()")
983@dcos_1_10
983984def test_private_repository_mesos_app():
984 """ Test private docker registry with mesos containerizer using "credentials" container field.
985 Note: Despite of what DC/OS docmentation states this feature is not yet implemented:
986 https://issues.apache.org/jira/browse/MESOS-7088
987 """
985 """ Test private docker registry with mesos containerizer using "config" container's image field."""
988986
989 client = marathon.create_client()
987 username = os.environ['DOCKER_HUB_USERNAME']
zen-dogUnsubmitted
Done

Those asserts here and in test_create_pod_with_private_image() are not necessary since you skip the test if they're not there anyway?

kensipeUnsubmitted
Not Done

yes

kensipeUnsubmitted
Not Done

I don't see docker_env_set() in the skipif.. however it is likely that we are skipping if we are missing expected ENV.. that makes these assert 'DOCKER_XYZ` not useful.

990 assert 'DOCKER_HUB_USERNAME' in os.environ, "Couldn't find docker hub username. $DOCKER_HUB_USERNAME is not set"
991 assert 'DOCKER_HUB_PASSWORD' in os.environ, "Couldn't find docker hub password. $DOCKER_HUB_PASSWORD is not set"
988 password = os.environ['DOCKER_HUB_PASSWORD']
992989
993 principal = os.environ['DOCKER_HUB_USERNAME']
994 secret = os.environ['DOCKER_HUB_PASSWORD']
990 secret_name = "dockerPullConfig"
991 secret_value_json = common.create_docker_pull_config_json(username, password)
995992
996 app_def = common.private_mesos_container_app(principal, secret)
997 client.add_app(app_def)
998 shakedown.deployment_wait()
993 import json
994 secret_value = json.dumps(secret_value_json)
999995
1000 common.assert_app_tasks_running(client, app_def)
996 client = marathon.create_client()
997 common.create_secret(secret_name, secret_value)
998
999 try:
1000 app_def = common.private_mesos_container_app(secret_name)
1001 client.add_app(app_def)
1002 shakedown.deployment_wait()
1003
1004 common.assert_app_tasks_running(client, app_def)
1005 finally:
1006 common.delete_secret(secret_name)
10011007
10021008
10031009def test_ping(marathon_service_name):
10041010 """ Tests the API end point for marathon /ping
10051011 This isn't provided by the client object and will need to create the url to test
10061012 """
10071013 response = common.http_get_marathon_path('ping', marathon_service_name)
10081014 assert response.status_code == 200
▲ Show 20 Lines • Show All 238 Lines • Show Last 20 Lines
View Options

tests/system/marathon_pods_tests.py

11"""Marathon pod acceptance tests for DC/OS."""
22
3import common
34import os
45import pytest
56import uuid
67import retrying
78import shakedown
89import time
910
1011from distutils.version import LooseVersion
1112from urllib.parse import urljoin
1213
1314from common import (block_port, cluster_info, event_fixture, get_pod_tasks, ip_other_than_mom,
1415 pin_pod_to_host, restore_iptables, save_iptables)
1516from dcos import marathon, util, http
16from shakedown import dcos_1_9, dcos_version_less_than, private_agents, required_private_agents
17from shakedown import dcos_1_9, dcos_1_10, dcos_version_less_than, private_agents, required_private_agents
1718from utils import fixture_dir, get_resource, parse_json
1819
1920
2021PACKAGE_NAME = 'marathon'
2122DCOS_SERVICE_URL = shakedown.dcos_service_url(PACKAGE_NAME) + "/"
2223
2324
2425def _pods_json(file="simple-pods.json"):
2526 return get_resource(os.path.join(fixture_dir(), file))
2627
2728
2829def _clear_pods():
2930 # clearing doesn't cause
3031 try:
3132 client = marathon.create_client()
3233 pods = client.list_pod()
3334 for pod in pods:
3435 client.remove_pod(pod["id"], True)
3536 shakedown.deployment_wait()
3637 except:
3738 pass
3839
39
4040def _pods_url(path=""):
4141 return "v2/pods/" + path
4242
4343
4444def _pod_status_url(pod_id):
4545 path = pod_id + "/::status"
4646 return _pods_url(path)
4747
Show All 35 Lines
8383 pod_json = _pods_json()
8484 pod_json["id"] = pod_id
8585 client.add_pod(pod_json)
8686 shakedown.deployment_wait()
8787 pod = client.show_pod(pod_id)
8888 assert pod is not None
8989
9090
91@pytest.mark.skipif("docker_env_set()")
92@dcos_1_10
93def test_create_pod_with_private_image():
94 username = os.environ['DOCKER_HUB_USERNAME']
95 password = os.environ['DOCKER_HUB_PASSWORD']
kensipeUnsubmitted
Done

do we need these assertions?

96
97 secret_name = "dockerPullConfig"
98 secret_value_json = common.create_docker_pull_config_json(username, password)
99
100 import json
101 secret_value = json.dumps(secret_value_json)
102
103 client = marathon.create_client()
104 common.create_secret(secret_name, secret_value)
105
106 try:
107 pod_def = common.private_docker_pod(secret_name)
108 client.add_pod(pod_def)
109 shakedown.deployment_wait()
110 pod = client.show_pod(pod_def["id"])
kensipeUnsubmitted
Done

why do we define the app json for secrets behind a function and for the longer pod json we do not?
I'm a fan of pulling this out so the test focuses on the setup and assertion.. but it isn't that big of a deal.

111 assert pod is not None
112 finally:
113 client.delete_secret(secret_name)
114
115
91116@dcos_1_9
92117@pytest.mark.usefixtures("event_fixture")
93118def test_event_channel_for_pods():
94119 """ Tests the Marathon event channnel specific to pod events.
95120 """
96121 client = marathon.create_client()
97122 pod_id = "/pod-create"
98123
▲ Show 20 Lines • Show All 332 Lines • Show Last 20 Lines
View Options

tests/system/test_marathon_on_marathon_ee.py

Show First 20 Lines • Show All 159 Lines • ▼ Show 20 Line(s)
160160 url = urljoin(dcos_url(), 'acs/api/v1/acls/dcos:superuser/users/{}'.format(MOM_EE_SERVICE_ACCOUNT))
161161 req = http.get(url)
162162 assert req.json()['array'][0]['url'] == '/acs/api/v1/acls/dcos:superuser/users/{}/full'.format(MOM_EE_SERVICE_ACCOUNT), "Service account permissions couldn't be set"
163163
164164
165165def ensure_secret(strict=False):
166166 if has_secret(MOM_EE_SECRET_NAME):
167167 delete_secret(MOM_EE_SECRET_NAME)
168 create_secret(MOM_EE_SECRET_NAME, MOM_EE_SERVICE_ACCOUNT, strict)
168 create_sa_secret(MOM_EE_SECRET_NAME, MOM_EE_SERVICE_ACCOUNT, strict)
169169 assert has_secret(MOM_EE_SECRET_NAME)
170170
171171
172172def ensure_docker_credentials():
173173 # Docker username and password should be passed as environment variables `DOCKER_HUB_USERNAME`
174174 # and `DOCKER_HUB_PASSWORD` (usually by jenkins)
175175 assert 'DOCKER_HUB_USERNAME' in os.environ, "Couldn't find docker hub username. $DOCKER_HUB_USERNAME is not set"
176176 assert 'DOCKER_HUB_PASSWORD' in os.environ, "Couldn't find docker hub password. $DOCKER_HUB_PASSWORD is not set"
Show All 24 Lines