HomeMesosphereNo notifications. 4 unresolved issues.

Package Marathon using sbt-native-packager
ClosedAll Users

Authored by jasongilanfarr on Apr 13 2017, 2:06 AM.

Details

Reviewers
aquamatthias
timcharper
jeschkies
zen-dog
jenkins
Commits
rMARATHONad83a9cea579: Disable host key checking :(
rMARATHON1a32502c6dcd: Use the scripts in the home directory for signing
rMARATHON0c5e86a91407: Upload the RPMs too
rMARATHONef914ecd01a9: Correct quoting in the SSH command
rMARATHON013171ec3ff8: Complete the cleanup
rMARATHON07bb07f5f267: Bump the integration test timeout to an hour
rMARATHON83854aac4cee: Add more docs, address review comments
rMARATHONa6e3051f4b8a: Package xz and figure out where the script is
rMARATHONf0778187434e: return this is super confusing
rMARATHON602875f75b8a: Correct the if statement
rMARATHON989c0c3197e9: Publish before posting results
rMARATHON5f31e5f1020b: Disable docs and keep debugging
rMARATHONf34bed9c16d6: Set pipefail, remove ls -laR
rMARATHONf75f393cc1c9: More packaging fun
rMARATHON3c54a2c0f697: PREVIEW: Package Marathon using sbt-native-packager
rMARATHON4c3d89101924: More ssh options
rMARATHONef89bcc660df: Tags need to remove the 'v' prefix
rMARATHONc053cc3b7430: Groovy is super annoying
rMARATHON895239173a35: Fix intellij's formatting, comment out a little more so testing works. don't…
rMARATHON1ef4ba845cbe: Try rsync instead of scp
rMARATHON2792bad58381: Discover everything in the home directory
rMARATHONc44ba1e4f658: PREVIEW: Package Marathon using sbt-native-packager
rMARATHONc69ddfa07206: Add information on how to test master and more docs
rMARATHON37e1b2f8d805: Build with NativePackager-1.2.0-M8
rMARATHON421f8bd8c3e4: Native for debian9 works
rMARATHONc9dc278db637: Groovy syntax is confusing
rMARATHON6a480804b1d5: Now with more actual packaging
rMARATHON1ab6927431a5: Forgot a star prefix
rMARATHONc3e8485792da: cd into the correct directory
rMARATHONca4636d9a109: Delete more unused/obsolete files and add some comments
rMARATHON03421d5a930c: Remove obsolete stuff too
rMARATHON19d4246f3c6d: Set Gittag
rMARATHON4702adccf898: Remove scp'd artifacts
rMARATHONf56631a6023c: Keep discovering
rMARATHONd615d68b5619: PREVIEW: Package Marathon using sbt-native-packager
rMARATHONcafe3f5e306e: Try to get our packages into a special home
rMARATHON94f16adc09dc: Package Marathon using sbt-native-packager
rMARATHON88b0a58402ad: Still trying
rMARATHON505b89038b75: Adapt _some_ of the changes into the jenkinsfiles
rMARATHON49377462941b: Try to find out where the packageworks
rMARATHON0d00bca09161: Start testing RPM publishing
rMARATHON6b2f5f993b68: Maybe rsync?
JIRA Issues
JIRA MARATHON-2584 Upload Tagged Native Packages to Package Server
JIRA MARATHON-2323 Upload Snapshot Packages to S3
JIRA MARATHON-1977 Upload Tar and Sha of Tagged Package to S3
JIRA MARATHON-1972 Upload Release Docker Image to Docker Hub
JIRA MARATHON-2052 Migrate sbt assembly to sbt native packager (for rpm and deb artifacts) for tagged packages
JIRA MARATHON-1812 Migrate sbt assembly to sbt native packager (for rpm and deb artifacts) for snapshot packages
JIRA MARATHON-7066 Migrate sbt assembly to sbt native packager
Summary
  • Package using sbt-native-packager instead of the custom FPM scripts:
    • We become super consistent and get all of the combinations "for free"
    • target/universal - universal zip and txz file with bin/lib and windows/unix scripts to do everything
    • target/packages/systemd-marathon-{gitVersionMappedToRpm}.rpm -> SystemD rpm (recent centos/rhel/fedora)
    • target/packages/systemd-marathon-{gitVersion}.deb -> SystemD deb (recent ubuntus/debians)
    • target/packages/upstart-marathon-{gitVersion}.deb (older ubuntus)
    • target/packages/sysvinit-marathon-{gitVersionMappedToRpm}.rpm -> SysVInit RPM (older centos/rhel/fedora)
    • target/packages/sysvinit-marathon-{gitVersion}.deb (older debians
  • remove Sbt-Assembly and all the fat jar stuff, its not a good way to do this.
  • consistently package/archive and make snapshot builds uploading an opt-in for reviews.
  • remove team city stuff, don't need it anymore.
  • add a packageAll command that will package up every supported package edition we want.
  • Remove a bunch of obsolete scripts.
  • Update the AMI
  • Add docs on how to test it.
  • Bump integration test timeout to 60 min.
Test Plan

Tested that /v2/leader works when the environment variables are set for:
Docker, debian+systemd, universal package, rpm+systemd. Don't really have access to all the older combos.
https://jenkins.mesosphere.com/service/jenkins/view/Marathon/job/phabricator-package-test-d696/ => specifically tests this change since it has a bunch of updates.

Tried to create an automated suite to test the different combos using... kvm, systemd-nspawn, docker, etc. This was a massive waste of time.

Tested the actual uploaded packages off of the package server.

Diff Detail

Repository
rMARATHON marathon
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
Changes from before your most recent comment are hidden. Show Older Changes
jeschkies accepted this revision.Apr 19 2017, 6:45 PM

As we've discussed. I'm fine if you fix install.bash and the simple s3 upload of the txz, zip and sha.

This revision is now accepted and ready to land.Apr 19 2017, 7:16 PM
jasongilanfarr marked 9 inline comments as done.
jasongilanfarr edited the test plan for this revision. (Show Details)
  • Remove AMI changes, separate review
  • Address all review comments.
  • Upload to the downloads.mesosphere.io bucket in the correct folders. This already has public access, so we don't have to do much more.
  • Add a note why we rsync then copy files around on the package server.
jenkins requested changes to this revision.Apr 19 2017, 8:29 PM
This revision now requires changes to proceed.Apr 19 2017, 8:29 PM
This revision is now accepted and ready to land.Apr 19 2017, 9:29 PM
This revision was automatically updated to reflect the committed changes.