Terraform-provider-aws: Add SageMaker support
Amazon has announced the new service SageMaker to build, train, and deploy machine learning models at scale.
Prerequisite
aws-sdk-go v1.12.36 (https://github.com/terraform-providers/terraform-provider-aws/pull/2474)
Affected Resource(s)
The following resources are needed.
For building model and training:
- aws_sagemaker_notebook_instance (https://github.com/terraform-providers/terraform-provider-aws/pull/2999)
- aws_presigned_notebook_instance_url
- aws_sagemaker_training_job (https://github.com/terraform-providers/terraform-provider-aws/pull/2955)
- aws_sagemaker_lifecycle_config resource (https://github.com/terraform-providers/terraform-provider-aws/pull/7585)
For the deployment/hosting part:
- aws_sagemaker_model (open PR is https://github.com/terraform-providers/terraform-provider-aws/pull/2478)
- aws_sagemaker_endpoint (open PR is https://github.com/terraform-providers/terraform-provider-aws/pull/2479)
- aws_sagemaker_endpoint_configuration (open PR is https://github.com/terraform-providers/terraform-provider-aws/pull/2477)
Expected Behavior
Create, update, delete, and import for above listed SageMaker resources of deployment part:
resource "aws_sagemaker_endpoint" "foo" {
name = "endpoint-foo"
endpoint_config_name = "${aws_sagemaker_endpoint_configuration.ec.name}"
}
resource "aws_sagemaker_endpoint_configuration" "ec" {
name = "endpoint-config-foo"
production_variants {
variant_name = "variant-1"
model_name = "${aws_sagemaker_model.m.name}"
initial_instance_count = 1
instance_type = "m3.xlarge"
initial_variant_weight = 1
}
}
resource "aws_sagemaker_model" "m" {
name = "my-model"
primary_container {
image = "111111111111.ecr.us-west-2.amazonaws.com/my-docker-image:latest"
model_data_url = "s3://111111111111-foo/model.tar.gz"
}
}
Example
A terraform example that shows how to use the above resources to deploy your own model is here: https://github.com/terraform-providers/terraform-provider-aws/pull/2585
References
jckuester
All 53 comments
Has anyone picked this one up yet?
darrenhaken
on 13 Dec 2017
@darrenhaken yes, I am working on it. See the linked PRs in the issue :-)
jckuester
on 13 Dec 2017
@jckuester I can't see a PR for the notebook instances, are you working on that too?
darrenhaken
on 15 Dec 2017
No, please start working on them if you like :)
jckuester
on 15 Dec 2017
Hi @jckuester @darrenhaken , anyone working on the training job resource? I'd like to contribute to this if possible
ddcprg
on 20 Dec 2017
@ddcprg I am only working on the linked PRs above (i.e. for deployment). So please contribute the training job resource if you like.
jckuester
on 3 Jan 2018
cool @jckuester ! I'll be taking a look soon and comment back on this ticket with an initial prototype
ddcprg
on 4 Jan 2018
@jckuester I now you've done this in your PR but I've raised #2924 just to add the vendor as per the README
I have the training job nearly done, I'm working on testing and I'll add the PR to this ticket as soon as I consider it ready
ddcprg
on 10 Jan 2018
Training Job PR -> #2955
If no one has picked up the notebook resource yet I'll take it
ddcprg
on 12 Jan 2018
Notebook Instance PR -> #2999
ddcprg
on 15 Jan 2018
How can we help to put some traction on these PR's? I'm going to go over the code and add any new features included by AWS recently if needed
ddcprg
on 16 May 2018
@jckuester would you mind adding a new item to the list above to include support for "lifecycle configurations"? I'll take a look and post the PR number later
ddcprg
on 16 May 2018
@randomcamel @radeksimko how can we help to push these PR's forward?
ddcprg
on 13 Jun 2018
+1, very interested in this functionality
Erstwild
on 13 Jul 2018
Not been any movement on this for a while (looks like failed integration tests), any chance of getting it pushed on?
ashleyjkell
on 20 Aug 2018
This has been open for long time, we are still waiting feedback from Hashicorp. The failures in the build are due to other issues in master at the time of merging, they may well have gone away by now
ddcprg
on 20 Aug 2018
@ashleyjkell from time to time, I rebased my PRs (https://github.com/terraform-providers/terraform-provider-aws/pull/2477 - https://github.com/terraform-providers/terraform-provider-aws/pull/2479) to make integration test green again (I guess they still are). But as long as there is no feedback coming from HashiCorp I will not wasting any more energy here to keeping PRs uptodate/resolving conflicts with main branch.
jckuester
on 21 Aug 2018
Any updates on this one?
Anyone from Hashicorp?
rainmanh
on 5 Oct 2018
I am interested in this functionality as well. Updates?
chasmosis
on 29 Oct 2018
I've just updated my PR's so they are green again. It'd be nice to have some feedback from @bflad or other TF committers. The PR's have been opened for about 10 months now
ddcprg
on 1 Nov 2018
@bflad Is there any way to get these PR's approved and merged?
bcatubig
on 5 Nov 2018
Also adding my voice to this one. Interested in getting this merged
t-lanigan
on 5 Nov 2018
Hello! I'm sorry for the delay and I'm actively looking at getting SageMaker support merged into the aws provider.
mbfrahry
on 8 Nov 2018
I've just finished my review of the instance and training job resources and will be moving down the list noted here shortly.
mbfrahry
on 8 Nov 2018
awesome @mbfrahry ! I'll try to go through you review today and make the necessary changes
ddcprg
on 9 Nov 2018
Thanks @mbfrahry! I'll try find to time on the weekend/next week to go through your reviews.
jckuester
on 15 Nov 2018
Hi! We're using Terraform v0.11.3. Do we need to update it to last Terraform version in order to use this new module (when merged) or it will get sync'd during terraform init (keeping the 0.11.3)?
reynico
on 27 Nov 2018
Hey @reynico, you won't have to do anything with modifying the version of Terraform. If you're pinned to a specific version of the aws provider, then you'll have to modify that to get these changes when they're released.
mbfrahry
on 27 Nov 2018
Any estimate at all when this will be released?
rainmanh
on 28 Nov 2018
Hello everyone. Unfortunately I've not been able to do the modifications as per @mbfrahry 's review and right now my access to AWS is limited and won't be able to test the code with the changes.
If anyone is willing to pick the first two bullets in the description up that'd be great, the PR is #2999 Perhaps @jckuester ? Whoever takes this on can squash my commits or start from scratch if you prefer.
ddcprg
on 4 Jan 2019
Hey @ddcprg, sorry to hear about your troubles. I'll take this on with @tracypholmes if it hasn't been picked up yet
mbfrahry
on 7 Jan 2019
Sure @mbfrahry pick it up, apologies for the inconveniences
ddcprg
on 8 Jan 2019
The new aws_sagemaker_notebook_instance resource has been released in version 1.56.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.
bflad
on 16 Jan 2019
That is a great milestone! I am sorry to be "that person", but I have been more interested in the 3 outstanding deployment and hosting PRs: 2477, 2478, and 2479. Is there a current ETA on those now too?
Erstwild
on 17 Jan 2019
@Erstwild we are very close :) I addressed all comments on the sagemaker PRs (which are in the 2nd feedback round now) and am now waiting for approval.
jckuester
on 21 Jan 2019
@julesjcraske
Is there any plan for implementing the ability of disabling the Direct internet access
And what about of attaching Lifecycle configurations ?
Many thanks
rainmanh
on 22 Jan 2019
I would say the following are required in a followup pr.
- LifecycleConfigName
- DirectInternetAccess
- DefaultCodeRepository
- AdditionalCodeRepositories
- VolumeSizeInGB (?)
- AcceleratorTypes (?)
bcatubig
on 22 Jan 2019
Hi there,
Do we have any specific roadmap or timelines by which we are certain about PRs: 2477, 2478, and 2479? Along with attaching Lifecycle configurations ?
saritajoshi9389
on 8 Feb 2019
The new aws_sagemaker_model resource has been released in version 1.58.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.
bflad
on 8 Feb 2019
Added a new PR for the aws_sagemaker_lifecycle_config resource (https://github.com/terraform-providers/terraform-provider-aws/pull/7585). I also have updates for the aws_sagemaker_notebook_instance in the pipeline (@saritajoshi9389, @bcatubig), as we need those resources in our company @yoyolabsio.
jckuester
on 15 Feb 2019
Added a PR for DirectInternetAccess support #7884 . Would love some feedback to see if things could be improved.
bcatubig
on 10 Mar 2019
Here is another PR (https://github.com/terraform-providers/terraform-provider-aws/pull/8011) that adds all missing attributes to aws_sagemaker_instance, ie. adapts the resource to the latest AWS API.
jckuester
on 19 Mar 2019
The aws_sagemaker_endpoint_configuration resource has been released in version 2.4.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.
bflad
on 29 Mar 2019
It would be really great to have the notebook lifecycle implemented. This is crucial for one of my clients.
gattil
on 4 Apr 2019
Hi folks 👋 The following will be releasing in version 2.5.0 of the Terraform AWS Provider shortly:
- New Resource:
aws_sagemaker_endpoint - New Resource:
aws_sagemaker_notebook_instance_lifecycle_configuration - resource/aws_sagemaker_notebook_instance: Add
lifecycle_config_nameargument
bflad
on 5 Apr 2019
@bflad Is this released?
rverma-nikiai
on 13 May 2019
@rverma-nikiai The resources available for the AWS provider can be found at https://www.terraform.io/docs/providers/aws/ - if you type 'sagemaker' in the search box at the top left the resources mentioned above (and the new argument) have indeed been released.
tdmalone
on 13 May 2019
Created a new PR for DirectInternetAccess support as I nuked my previous PR by mistake. The new PR is https://github.com/terraform-providers/terraform-provider-aws/pull/8618
bcatubig
on 13 May 2019
Is it possible to change the timeout for creating the sagemaker endpoint? For me it stops after 1m saying
ResourceNotReady: failed waiting for successful resource state
Usually creating the endpoint I have takes around 15 minutes. Adding
timeouts {
create = "30m"
delete = "30m"
}
like it is possible on other resources gives me:
Error decoding timeout: Timeout Key (create) is not supported
LarsNeR
on 1 Aug 2019
Hi @LarsNeR 👋 My recommendation would be to open submit new GitHub issues following the bug report and feature request templates for existing functionality. These large "support service X" GitHub issues tend to not have a clear definition of done and usually get closed out in preference of more targeted issues for tracking individual asks.
bflad
on 1 Aug 2019
I've just seen it was an error on my site. Works as expected now.
LarsNeR
on 1 Aug 2019
Hi folks 👋
We just merged direct_internet_access argument support into the aws_sagemaker_notebook_instance resource. It will release with version 2.47.0 of the Terraform AWS Provider, tomorrow. Thanks to @bcatubig for the implementation. 👍
Please note that I'm going to close this GitHub issue since the majority of the original work for it has been completed and since there is not a clear definition of done over time. If you have specific bug reports with existing functionality or feature requests for additional functionality please use those issue templates. Thanks.
bflad
on 29 Jan 2020
I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.
If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!
![hashibot[bot] picture](https://avatars2.githubusercontent.com/in/8332?v=4&s=40)
hashibot[bot]
on 27 Mar 2020
Related issues
reedloden
·
3Comments
hashibot
·
3Comments
hashibot
·
3Comments
modax
·
3Comments
hashibot
·
3Comments
Most helpful comment
Any estimate at all when this will be released?