Terraform-provider-aws: Feature Request: Managed Streaming for Kafka

I have a basic implementation of this feature working in PR #6809. This is my first contribution to this project, so I am not sure what the acceptance criteria for PRs is.

One issue with MSK right now is Update operations are not yet implemented in the AWS MSK api. I am unsure how to handle updating resources in this case? Seems dangerous to ForceNew for all changes to the resource.

@jrefi any updates? I'm looking forward for this function. Can I help you?

@random1st Sorry, got caught up in my day to day work. Just fixed the unit tests for my MR. I am not quite sure what all remains, so if you don't mind glancing at it, that would be great.

@jrefi

1. Ask to reformat code make fmt
2. Wrong cluster name test_cluster - valid is like "test-cluster"
3. "Broker node group information is invalid. Security groups list is empty. See https://docs.aws.amazon.com/en_us/msk/latest/developerguide/create-cluster.html - it is required param

@random1st

1. Ran that command resulted in no code changes.
2. Why is test_cluster invalid?
3. Made security groups required.

@jrefi test_cluster is invalid because underscore is not supported

@random1st Where is that specified? Seems odd, as I am seeing underscores used extensively for Terraform resource names.

resource "aws_msk_cluster" "test_cluster" { name = "%s" broker_count = 3 broker_instance_type = "kafka.m5.large" broker_volume_size = 10 broker_security_groups =["${aws_security_group.test_sg_a.id}"] kafka_version = "1.1.1" client_subnets = ["${aws_subnet.test_subnet_a.id}", "${aws_subnet.test_subnet_b.id}", "${aws_subnet.test_subnet_c.id}"] }

@jrefi

@jrefi amazing work! I think maybe if you just hardcode "test-cluster" instead of using "%s", that's all that's needed? Would be so happy to be able to use this. If there's a chance I can help move this over the line, let me know!

@dthtvwls Thanks. Yeah Id be open to help with whatever is left.

I am still confused why I need to change that test. I am passing in a cluster name so Im not sure what hard-coding would accomplish.

You know, it seems I mistakenly inferred that you were passing in _test_cluster_ as a name, which MSK would not have accepted, but a name like _terraform-msk-test-00000_ should be fine.

@random1st would you mind verifying again? I'm very eager to get this merged if possible and it looks so painfully close.

@dthtvwls I've checked an example configuration. Everything is ok. Let's fix the markdown description of example usage and merge it. broker_security_groups is not optional param and test_cluster is invalid name.

resource "aws_msk_cluster" "test_cluster" { name = "test_cluster" broker_count = 3 broker_instance_type = "kafka.m5.large" broker_volume_size = 10 broker_security_groups =["${aws_security_group.test_sg_a.id}"] kafka_version = "1.1.1" client_subnets = ["${aws_subnet.test_subnet_a.id}", "${aws_subnet.test_subnet_b.id}", "${aws_subnet.test_subnet_c.id}"] }

Trace:

2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: GET /v1/clusters?clusterNameFilter=test_cluster HTTP/1.1 2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: Host: kafka.us-east-1.amazonaws.com 2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: User-Agent: aws-sdk-go/1.16.19 (go1.10.4; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.9-beta1 2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: Authorization: AWS4-HMAC-SHA256 Credential=AKIAIT6SVQ2D6HAOBIFQ/20190314/us-east-1/kafka/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=31473150a38a3cd4b8d4f5dbdc0e4fb3a27ee0cf35f36588170a51f6e4ecd6e7 2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: Content-Type: application/x-amz-json-1.1 2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: X-Amz-Date: 20190314T072016Z 2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: Accept-Encoding: gzip 2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: 2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: 2019-03-14T10:20:16.032+0300 [DEBUG] plugin.terraform-provider-aws: ----------------------------------------------------- 2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: 2019/03/14 10:20:16 [DEBUG] [aws-sdk-go] DEBUG: Response Kafka/ListClusters Details: 2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: ---[ RESPONSE ]-------------------------------------- 2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: HTTP/2.0 400 Bad Request 2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: Content-Length: 124 2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: Content-Type: application/json 2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: Date: Thu, 14 Mar 2019 07:20:16 GMT 2019-03-14T10:20:16.675+0300 [DEBUG] plugin.terraform-provider-aws: Via: 1.1 029f15a661be82d29f31e88713b71d65.cloudfront.net (CloudFront)

@jrefi Please, let's fix it asap.
https://github.com/jrefi/terraform-provider-aws/pull/2

@random1st Merged

There is a conflict with root repository

hello, its great to see that you have already developing support for MSK. Can I get some estimation when would it be released?

@random1st Resolved

Guys, when can we merge it to master?

Guys, any plans on delivering MSK?

I believe my MR is about ready to go. If there is anything else I need to add, someone please let me know. (https://github.com/terraform-providers/terraform-provider-aws/pull/6809)

My organization has actually decided to transition away from MSK to self-hosted Kafka, so this work is no longer a priority for us. I am happy to get it across the finish line, but I don't have the bandwidth to hunt down approvers and get them to merge this in.

@mrf @tomelliff Can we get any guidance on what's needed for final reviews of this resource? #6809

Additionally, thank you for all your progress @jrefi. I and I'm sure many others are eagerly awaiting this functionality

+1 for #6809 it would be nice to have the implementation soon.

@hExPY Unfortunately I'm having little success in encouraging reviewers to check this code, even though it passes the tests and seems to be ready for a merge. The more people that show interest, the better chance we have!

@kesensoy in that case, allow me to pile on and +1 this as well!

@bploetz me too! +1

If @bploetz is in, i'm in. +1

Can’t wait 👍

Looking forward to this!!!! What does it take to get this merged already!?

+1

+1

@jrefi Great work! Can't wait to see it merged in master.

Posted a tweet with the hope more people will get involved and the PR will get the needed attention.

/offtopic
@jrefi Mind if I ask why your organization moved away from MSK in respect to self-hosted Kafka? In case you are allowed disclose such information.

Waiting for merge

Thanks so much for the great work! I really look forward to seeing this merged 🚀

Please see #8357 which supersedes the previous PR #6809

A new aws_msk_cluster resource has been merged and will release with version 2.12.0 of the Terraform AWS Provider, likely tomorrow. Special thanks to @jrefi, @dthtvwls, and @jesseaukeman for making this possible! 🚀

For future bug reports and feature requests with MSK, please submit new GitHub issues. Thanks!

This has been released in version 2.12.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

This has been released in version 2.12.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

Hi , I have tried creating AWS MSK cluster with reference from the document shared. But I am getting the error "Error: error creating MSK cluster: BadRequestException: The parameter value contains one or more characters that are not valid.
status code: 400, request id: e2589e6a-8161-11e9-8f31-6f8877605e30". Below is the terraform code used. Please help me to fix the error.

resource "aws_msk_cluster" "msk_cluster" {
cluster_name = "Testing_Cluster"
kafka_version = "2.1.0"
number_of_broker_nodes = 3

broker_node_group_info {
instance_type = "kafka.m5.large"
client_subnets = [
"${aws_subnet.subnet_a.id}",
"${aws_subnet.subnet_b.id}",
"${aws_subnet.subnet_c.id}",
]
ebs_volume_size = 5
security_groups = [ "${aws_security_group.MSK_Sg.id}" ]
}

tags = {
Name = "Cluster_MSK"
}
}

Hi @navinsnn53 :wave: in general please submit a new GitHub issue so the maintainers and community can appropriately triage your issue instead of commenting on a closed issue, which may get lost/locked over time.

That said, my hunch would be that the MSK service does not allow the usage of underscores (_) in cluster names. The MSK API Reference, does not provide too much guidance for string validation on fields such as the cluster name. We currently only implement the known string length check within the Terraform resource for the cluster_name argument: https://github.com/terraform-providers/terraform-provider-aws/blob/5619c0e0ac94f4dd645dd2f388c19795b5800341/aws/resource_aws_msk_cluster.go#L83

Locking to encourage new GitHub issues.