Skip to content

bump terraform providers - #83279

Merged
flokli merged 3 commits into
NixOS:masterfrom
flokli:bump-terraform-providers
Apr 2, 2020
Merged

bump terraform providers#83279
flokli merged 3 commits into
NixOS:masterfrom
flokli:bump-terraform-providers

Conversation

@flokli

@flokli flokli commented Mar 24, 2020

Copy link
Copy Markdown
Member
Motivation for this change

This updates the terraform provider update script (thanks @zimbatm for helping out here), and updates all providers.

I haven't finished rebuilding all providers yet.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@flokli
flokli requested review from alexfmpe, kalbasit and zimbatm March 24, 2020 11:50
zimbatm
zimbatm reviewed Mar 24, 2020
View reviewed changes
Comment thread pkgs/applications/networking/cluster/terraform-providers/data.nix Outdated
@flokli

flokli commented Mar 24, 2020

Copy link
Copy Markdown
Member Author

terraform-providers.google and terraform-providers.google-beta fail to build:

❯ nix-build -A terraform-providers.google
these derivations will be built:
  /nix/store/ml84kwg898dfy9909cpnz5h428651d8q-terraform-provider-google-3.14.0.drv
building '/nix/store/ml84kwg898dfy9909cpnz5h428651d8q-terraform-provider-google-3.14.0.drv'...
unpacking sources
unpacking source archive /nix/store/jjxmsxl0haxj77xf2jd78cxwpaqp211i-source
source root is source
patching sources
configuring
building
go/src/github.com/terraform-providers/terraform-provider-google/google/config.go:51:2: code in directory /build/go/src/github.com/terraform-providers/terraform-provider-google/vendor/google.golang.org/api/sqladmin/v1beta4 expects import "google.golang.org/api/sql/v1beta4"
builder for '/nix/store/ml84kwg898dfy9909cpnz5h428651d8q-terraform-provider-google-3.14.0.drv' failed with exit code 1
error: build of '/nix/store/ml84kwg898dfy9909cpnz5h428651d8q-terraform-provider-google-3.14.0.drv' failed
❯ nix-build -A terraform-providers.google-beta
these derivations will be built:
  /nix/store/1l01hi54y4glw11hi9yb756ijkx4j8v0-terraform-provider-google-beta-3.14.0.drv
building '/nix/store/1l01hi54y4glw11hi9yb756ijkx4j8v0-terraform-provider-google-beta-3.14.0.drv'...
unpacking sources
unpacking source archive /nix/store/dij55sqlwxrpr98ak99w6vnw6m373qif-source
source root is source
patching sources
configuring
building
go/src/github.com/terraform-providers/terraform-provider-google-beta/google-beta/config.go:52:2: code in directory /build/go/src/github.com/terraform-providers/terraform-provider-google-beta/vendor/google.golang.org/api/sqladmin/v1beta4 expects import "google.golang.org/api/sql/v1beta4"
builder for '/nix/store/1l01hi54y4glw11hi9yb756ijkx4j8v0-terraform-provider-google-beta-3.14.0.drv' failed with exit code 1
error: build of '/nix/store/1l01hi54y4glw11hi9yb756ijkx4j8v0-terraform-provider-google-beta-3.14.0.drv' failed

There seems to be something wrong with the vendored libraries I guess?

@flokli

flokli commented Mar 24, 2020

Copy link
Copy Markdown
Member Author

When checking out and building the provider manually, there seems to be some differing behaviour:

$ GO111MODULE=off go build
google/config.go:51:2: code in directory /home/zimbatm/go/src/github.com/terraform-providers/terraform-provider-google/vendor/google.golang.org/api/sqladmin/v1beta4 expects import "google.golang.org/api/sql/v1beta4"

(fails)

$ GO111MODULE=on go build -mod=vendor

(succeeds)

@flokli
flokli force-pushed the bump-terraform-providers branch from 05f599b to beb2c00 Compare March 24, 2020 19:09
@flokli

flokli commented Mar 26, 2020

Copy link
Copy Markdown
Member Author

I opened a support ticket at GitHub about the fact that some repositories are missing from the API:

to reproduce, look for terraform-provider-{dns,external,template} in

hub api --paginate https://api.github.com/orgs/terraform-providers/repos | jq | vim -

Or if you want to go fancy, GraphQL:

$ hub api --paginate graphql -f query='
  query($endCursor: String) {
    repositoryOwner(login: "terraform-providers") {
      repositories(first: 100, after: $endCursor) {
        nodes {
          nameWithOwner
          name
          refs(first: 1, refPrefix: "refs/tags/", orderBy: {field: TAG_COMMIT_DATE, direction: DESC}) {
            nodes {
              name                                                                                                                                                                                                                                 
            }                                                                                                                                                                                                                                      
          }                                                                                                                                                                                                                                        
        }                                                                                                                                                                                                                                          
        pageInfo {                                                                                                                                                                                                                                 
          hasNextPage                                                                                                                                                                                                                              
          endCursor                                                                                                                                                                                                                                
        }                                                                                                                                                                                                                                          
      }                                                                                                                                                                                                                                            
    }                                                                                                                                                                                                                                              
  }                                                                                                                                                                                                                                                
' | jq | vim -

@flokli

flokli commented Mar 26, 2020

Copy link
Copy Markdown
Member Author

Alright, already heard back. That was quick! :-)

Turns out, these repos have been moved to the hashicorp organization, and we were fooled by the redirect - so we might want to look at the hashicorp organization too, and filter for terraform-provider-* repos to include these…

@flokli
flokli force-pushed the bump-terraform-providers branch from beb2c00 to 5ef47a2 Compare March 30, 2020 10:48
@flokli

flokli commented Mar 30, 2020

Copy link
Copy Markdown
Member Author

I pushed a new version, that uses hub, the GraphQL endpoint, and fetches from both hashicorp and terraform-providers.

This required some refactoring, but now we should have back all terraform providers and not run into GitHubs ratelimiting, as we're not generating a HTTP request for each provider.

PTAL.

@flokli
flokli force-pushed the bump-terraform-providers branch 2 times, most recently from 6a1550f to 3123e88 Compare March 30, 2020 10:53
zimbatm
zimbatm approved these changes Mar 30, 2020
View reviewed changes
Comment thread pkgs/applications/networking/cluster/terraform-providers/update-all Outdated
@flokli
flokli force-pushed the bump-terraform-providers branch 4 times, most recently from 42c7d97 to af94137 Compare March 30, 2020 11:19
@ofborg ofborg Bot added 8.has: clean-up This PR removes packages or removes other cruft 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. labels Mar 30, 2020
@flokli
flokli force-pushed the bump-terraform-providers branch from af94137 to bdbb2a7 Compare March 31, 2020 14:00
@flokli

flokli commented Mar 31, 2020

Copy link
Copy Markdown
Member Author

I bumped the terraform providers once more, now including the fix for hashicorp/terraform-provider-google#5970.

@flokli
flokli requested a review from zimbatm March 31, 2020 14:02
zimbatm
zimbatm reviewed Mar 31, 2020
View reviewed changes
Comment thread pkgs/applications/networking/cluster/terraform-providers/data.nix Outdated
@flokli
flokli force-pushed the bump-terraform-providers branch from bdbb2a7 to c683e86 Compare April 2, 2020 12:19
@ofborg ofborg Bot added 10.rebuild-linux: 101-500 This PR causes between 101 and 500 packages to rebuild on Linux. and removed 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. labels Apr 2, 2020
@flokli

flokli commented Apr 2, 2020 •
edited
Loading

Copy link
Copy Markdown
Member Author

Some providers that were moved to the hashicorp organization still have references to the terraform-providers organization, which needs patching. Working on this now.

flokli and others added 2 commits April 2, 2020 17:53
@flokli @zimbatm
terraform-providers: refactor
2e19bc7 
Some terraform providers were moved to the `hashicorp` organization,
some are still in the `terraform-providers` organization.

GitHub doesn't show moved repositories when querying via the API (only
on the web interface). In addition, if there are more than 100 results
(there still are), we'd need to implement paging, which is cubersome in curl.

In addition to that, to look up tags, we used to creating a HTTP request
for each and every provider, which caused us to easily get blocked by
GitHubs rate-limiting.

Instead of all that, let's use `hub` (which implements paging) and the
GitHub's GraphQL endpoint to get a list all repositories for a given
organization AND the latest tag for each of them.

We post-process the output with `jq`.

This also removes the providers.txt and folds the data into the script.
The blacklist was becoming too annoying to handle as a config file.

For the whitelist, we still need to individually fetch tags.

Fixes NixOS#83215

Co-Authored-By: zimbatm <zimbatm@zimbatm.com>
@zimbatm @flokli
terraform-providers: fix the google and google-beta providers
3e5149a
@flokli
flokli force-pushed the bump-terraform-providers branch from c683e86 to 3090ef7 Compare April 2, 2020 16:28
@flokli

flokli commented Apr 2, 2020 •
edited
Loading

Copy link
Copy Markdown
Member Author

I sent PRs to all of the providers that were renamed:

and added workarounds in special-providers for the time being.

I disabled the broken terraform providers

  • terraform-providers/terraform-provider-circonus
  • terraform-providers/terraform-provider-vmc

Apart from that, I could build terraform-full and terraform_0_11-full.

@flokli
flokli force-pushed the bump-terraform-providers branch from 3090ef7 to 9f270c6 Compare April 2, 2020 16:33
@flokli

flokli commented Apr 2, 2020

Copy link
Copy Markdown
Member Author

@GrahamcOfBorg build terraform-full terraform_0_11-full

@flokli
flokli requested a review from edef1c April 2, 2020 16:35
@flokli flokli mentioned this pull request Apr 2, 2020
Merged
10 tasks
@ofborg ofborg Bot added 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. and removed 10.rebuild-linux: 101-500 This PR causes between 101 and 500 packages to rebuild on Linux. labels Apr 2, 2020
@flokli
flokli merged commit 26afaba into NixOS:master Apr 2, 2020
@flokli
flokli deleted the bump-terraform-providers branch April 2, 2020 17:00
This was referenced Jul 6, 2020
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zimbatm zimbatm zimbatm approved these changes
@kalbasit kalbasit Awaiting requested review from kalbasit
@alexfmpe alexfmpe Awaiting requested review from alexfmpe
@edef1c edef1c Awaiting requested review from edef1c

Assignees

No one assigned

Labels

8.has: clean-up This PR removes packages or removes other cruft 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@flokli @zimbatm