HomeMesosphereNo notifications. 4 unresolved issues.
  • Search
Differential D724

Provide S3 Credentials via the default AWS provider chain.
ClosedAll Users
Actions

Authored by aquamatthias on Apr 25 2017, 12:32 AM.

Details

Reviewers
jenkins
jeschkies
• jasongilanfarr
Commits
rSDKCORE82dbc83eca47: Provide S3 Credentials via the default AWS provider chain.
rMARATHON946784b988cc: Provide S3 Credentials via the default AWS provider chain.
JIRA Issues
JIRA MARATHON-7235 S3 Credentials are too restrictive
Summary

Use the AWS java client to retrieve credentials from the default provider chain.
This results in the following provider chain:

  • use credentials provided from URI parameters
  • use credentials set via the environment
  • use credentials set via system properties
  • use default credentials set via the credentials file
  • use credentials provided via the Amazon EC2 Container Service
  • use credential defined via system configuration in akka.stream.alpakka.s3
Test Plan

Backup to an S3 storage without providing crentials:
runMain mesosphere.marathon.core.storage.backup.Backup --backup_location s3://bucket/file?region=eu-central-1

Diff Detail

Repository
rMARATHON marathon
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
Changes from before your most recent comment are hidden. Show Older Changes
jeschkies requested changes to this revision.Apr 25 2017, 12:04 PM

Thanks. I just have two questions.

src/main/scala/mesosphere/marathon/stream/UriIO.scala
110–113

I'm wondering if we should say that we first check the uri parameters and then follow the default provider chain as documented here.

Is providing the credentials in the URI save?

This revision now requires changes to proceed.Apr 25 2017, 12:04 PM
• jasongilanfarr accepted this revision.Apr 25 2017, 4:30 PM

Could you also add the reference.conf section as I suggested in the bug?

aquamatthias added inline comments.Apr 26 2017, 1:50 PM
src/main/scala/mesosphere/marathon/stream/UriIO.scala
110–113

This comment describes the chain to get credentials, by also describing how the AWS default chain works.
Do you want me to change this?

The main focus of this PR is to enable credential providers, so it is not necessary to send them in the URL.

jeschkies accepted this revision.Apr 26 2017, 4:29 PM
This revision is now accepted and ready to land.Apr 26 2017, 4:29 PM
Closed by commit rMARATHON946784b988cc: Provide S3 Credentials via the default AWS provider chain. (authored by aquamatthias). · Explain WhyApr 26 2017, 4:31 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

Diff 2911

View Options

docs/docs/command-line-flags.md

Show First 20 Lines • Show All 163 Lines • ▼ Show 20 Line(s)
164164* <span class="label label-default">v1.5.0</span>`--minimum_viable_task_execution_duration` (Optional. Default: 60 seconds):
165165 Delay (in ms) after which a task is considered viable. If the task starts up correctly, but fails during this timeout, the application is backed off.
166166* <span class="label label-default">v1.5.0</span>`--backup_location` (Optional. Default: None):
167167 Create a backup before a migration is applied to the persistent store.
168168 This backup can be used to restore the state at that time.
169169 Currently two providers are allowed:
170170 - File provider: file:///path/to/file
171171 - S3 provider (experimental): s3://bucket-name/key-in-bucket?access_key=xxx&secret_key=xxx&region=eu-central-1
172
172 Please note: access_key and secret_key are optional.
173 If not provided, the [AWS default credentials provider chain](http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) is used to look up aws credentials.
173174
174175## Tuning Flags for Offer Matching/Launching Tasks
175176
176177Mesos frequently sends resource offers to Marathon (and all other frameworks). Each offer will represent the
177178available resources of a single node in the cluster. Before this <span class="label label-default">v0.8.2</span>,
178179Marathon would only start a single task per
179180resource offer, which led to slow task launching in smaller clusters.
180181
▲ Show 20 Lines • Show All 171 Lines • Show Last 20 Lines
View Options

docs/docs/rest-api/public/api/v2/leader.raml

Show All 29 Lines
3030 Every abdication triggers a new leader election.
3131 The next elected leader will read the state from the persistent store and continue the work from the previous leader.
3232
3333 It is possible to use this endpoint to trigger a backup or restore operation for the persistent data store.
3434 The requested operation will be perfomed by the next leading Marathon master.
3535 Currently two providers are allowed:
3636 - File provider: file:///path/to/file
3737 - S3 provider (experimental): s3://bucket-name/key-in-bucket?access_key=xxx&secret_key=xxx&region=eu-central-1
38 Please note: access_key and secret_key are optional.
39 If not provided, the [AWS default credentials provider chain](http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) is used to look up aws credentials.
40
3841
3942 queryParameters:
4043 backup?:
4144 required: false
4245 description: URI to backup zk state.
4346 example: file:///var/backup/marathon/backup
4447 restore?:
4548 required: false
Show All 19 Lines
View Options

project/Dependencies.scala

Show All 17 Lines
1818
1919 val marathon = (Seq(
2020 // runtime
2121 akkaActor % "compile",
2222 akkaSlf4j % "compile",
2323 akkaStream % "compile",
2424 akkaHttp % "compile",
2525 asyncAwait % "compile",
26 aws % "compile",
2627 chaos % "compile",
2728 mesos % "compile",
2829 jodaTime % "compile",
2930 jodaConvert % "compile",
3031 jerseyServlet % "compile",
3132 jerseyMultiPart % "compile",
3233 jettyEventSource % "compile",
3334 uuidGenerator % "compile",
Show All 38 Lines
7273 val benchmark = Seq(
7374 Test.jmh
7475 )
7576}
7677
7778object Dependency {
7879 object V {
7980 // runtime deps versions
80 val Alpakka = "0.6"
81 val Aws = "1.11.123"
82 val Alpakka = "0.7"
8183 val Chaos = "0.8.8"
8284 val Guava = "19.0"
8385 val Mesos = "1.2.0"
8486 // Version of Mesos to use in Dockerfile.
8587 val MesosDebian = "1.2.0-2.0.6"
8688 val OpenJDK = "openjdk:8u121-jdk"
8789 val Akka = "2.4.17"
8890 val ApacheCommonsCompress = "1.13"
Show All 26 Lines
115117 val JUnitBenchmarks = "0.7.2"
116118 val JMH = "1.14"
117119 val ScalaCheck = "1.13.4"
118120 }
119121
120122 val excludeMortbayJetty = ExclusionRule(organization = "org.mortbay.jetty")
121123 val excludeJavaxServlet = ExclusionRule(organization = "javax.servlet")
122124
125 val aws = "com.amazonaws" % "aws-java-sdk-core" % V.Aws
123126 val alpakkaS3 = "com.lightbend.akka" %% "akka-stream-alpakka-s3" % V.Alpakka
124127 val akkaActor = "com.typesafe.akka" %% "akka-actor" % V.Akka
125128 val akkaSlf4j = "com.typesafe.akka" %% "akka-slf4j" % V.Akka
126129 val akkaStream = "com.typesafe.akka" %% "akka-stream" % V.Akka
127130 val akkaHttp = "com.typesafe.akka" %% "akka-http" % "10.0.5"
128131 val akkaHttpPlayJson = "de.heikoseeberger" %% "akka-http-play-json" % "1.10.1"
129132 val asyncAwait = "org.scala-lang.modules" %% "scala-async" % V.AsyncAwait
130133 val playJson = "com.typesafe.play" %% "play-json" % V.PlayJson
▲ Show 20 Lines • Show All 60 Lines • Show Last 20 Lines
View Options

src/main/scala/mesosphere/marathon/core/storage/backup/Backup.scala

Show All 18 Lines
1919
2020/**
2121 * Base class for backup and restore command line utility.
2222 */
2323abstract class BackupRestoreAction extends StrictLogging {
2424
2525 class BackupConfig(args: Seq[String]) extends ScallopConf(args) with StorageConf with NetworkConf {
2626 override def availableFeatures: Set[String] = Set.empty
27 override lazy val defaultNetworkName = opt[String]()
28 override lazy val mesosBridgeName = opt[String]()
2927 verify()
3028 require(backupLocation.isDefined, "--backup_location needs to be defined!")
3129 }
3230
3331 /**
3432 * Can either run a backup or restore operation.
3533 */
3634 @SuppressWarnings(Array("AsInstanceOf"))
▲ Show 20 Lines • Show All 62 Lines • Show Last 20 Lines
View Options

src/main/scala/mesosphere/marathon/stream/UriIO.scala

11package mesosphere.marathon
22package stream
33
44import java.net.URI
55import java.nio.file.Paths
66
77import akka.actor.ActorSystem
88import akka.http.scaladsl.model.ContentTypes
99import akka.stream.Materializer
10import akka.stream.alpakka.s3.S3Settings
1011import akka.stream.alpakka.s3.acl.CannedAcl
1112import akka.stream.alpakka.s3.auth.AWSCredentials
1213import akka.stream.alpakka.s3.impl.MetaHeaders
1314import akka.stream.alpakka.s3.scaladsl.S3Client
1415import akka.stream.scaladsl.{ FileIO, Source, Sink => ScalaSink }
1516import akka.util.ByteString
1617import akka.{ Done, NotUsed }
18import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
1719import com.typesafe.scalalogging.StrictLogging
1820import com.wix.accord.Validator
1921import com.wix.accord.dsl._
2022import mesosphere.marathon.api.v2.Validation.{ isTrue, uriIsValid }
2123
2224import scala.concurrent.{ ExecutionContext, Future }
25import scala.util.Try
2326
2427/**
2528 * UriIO provides sources from and sinks to an URI.
2629 * It supports different providers, so multiple schemes are supported.
2730 *
2831 * Provided schemes:
2932 *
3033 * File
▲ Show 20 Lines • Show All 64 Lines • ▼ Show 20 Line(s)
9598 case "file" if nonEmpty(uri.getPath) && uri.getPath.length > 1 => true
9699 case "s3" if nonEmpty(uri.getHost) && nonEmpty(uri.getPath) => true
97100 case _ => false
98101 }
99102 }
100103
101104 def valid: Validator[String] = uriIsValid and isTrue[String]{ uri: String => s"Invalid URI or unsupported scheme: $uri" }(uri => isValid(new URI(uri)))
102105
106 /**
107 * Create S3 client.
108 * The credentials use the following chain:
109 * - use credentials provided from URI parameters
110 * - use credentials set via the environment
111 * - use credentials set via system properties
112 * - use default credentials set via the credentials file
113 * - use credentials provided via the Amazon EC2 Container Service
jeschkiesUnsubmitted
Not Done

I'm wondering if we should say that we first check the uri parameters and then follow the default provider chain as documented here.

Is providing the credentials in the URI save?

aquamatthiasAuthorUnsubmitted
Not Done

This comment describes the chain to get credentials, by also describing how the AWS default chain works.
Do you want me to change this?

The main focus of this PR is to enable credential providers, so it is not necessary to send them in the URL.

114 * - use credential defined via system configuration in akka.stream.alpakka.s3
115 * @return The S3Client for the defined URI.
116 */
103117 private[this] def s3Client(uri: URI)(implicit actorSystem: ActorSystem, materializer: Materializer): S3Client = {
104118 val params = parseParams(uri)
105119 val region = params.getOrElse("region", "us-east-1")
106 val accessKey = params.getOrElse("access_key", "")
107 val accessSecret = params.getOrElse("secret_key", "")
108 logger.info(s"S3 settings region: $region accessKey:$accessKey accessSecret:$accessSecret")
109 val credentials = AWSCredentials(accessKey, accessSecret)
120 val credentials = {
121 def fromURL: Option[AWSCredentials] = for {
122 accessKey <- params.get("access_key")
123 accessSecret <- params.get("secret_key")
124 } yield AWSCredentials(accessKey, accessSecret)
125 def fromProviderChain: Option[AWSCredentials] = {
126 Try(new DefaultAWSCredentialsProviderChain().getCredentials)
127 .toOption
128 .map(creds => AWSCredentials(creds.getAWSAccessKeyId, creds.getAWSSecretKey))
129 }
130 fromURL.orElse(fromProviderChain).getOrElse(S3Settings(actorSystem).awsCredentials)
131 }
110132 new S3Client(credentials, region)
111133 }
112134
113135 private[this] def parseParams(uri: URI): Map[String, String] = {
114136 Option(uri.getQuery).getOrElse("").split("&").collect { case QueryParam(k, v) => k -> v }(collection.breakOut)
115137 }
116138
117139 private[this] object QueryParam {
118140 def unapply(str: String): Option[(String, String)] = str.split("=") match {
119141 case Array(key: String, value: String) => Some(key -> value)
120142 case _ => None
121143 }
122144 }
123145}