github/pre-commit-opentofu

Fork 1

mirror of https://github.com/tofuutils/pre-commit-opentofu.git synced 2025-10-15 17:38:54 +02:00

Maksym Vlasov 3d5a882a53

docs: Clarify docs for terraform_tfsec hook (#266 )

2021-10-26 14:35:55 +02:00

26 KiB

Raw Blame History

Collection of git hooks for Terraform to be used with pre-commit framework

Want to Contribute? Check open issues and contributing notes.

How to install
Available Hooks
Hooks usage notes and examples
Authors
License

How to install

1. Install dependencies

pre-commit, _{^{terraform,
_{^{git,
_{^{POSIX compatible shell,
_{^{Internet connection (on first run),
_{^{x86_64 compatible operation system,
_{^{Some hardware where this OS will run,
_{^{Electricity for hardware and internet connection,
_{^{Some basic physical laws,
_{^{Hope that it all will works.}}}}}}}}}}}}}}}}}}
checkov required for checkov hook.
terraform-docs required for terraform_docs hooks.
terragrunt required for terragrunt_validate hook.
terrascan required for terrascan hook.
TFLint required for terraform_tflint hook.
TFSec required for terraform_tfsec hook.
infracost required for infracost_breakdown hook.
jq required for infracost_breakdown hook.

Docker

If no --build-arg is specified, then the latest versions of pre-commit and terraform will be installed.

git clone git@github.com:antonbabenko/pre-commit-terraform.git
cd pre-commit-terraform
# Install all tools with latest versions:
docker build -t pre-commit --build-arg INSTALL_ALL=true .

You can specify needed tool versions by providing --build-arg's.
If you'd like you can use the latest versions:

docker build -t pre-commit \
    --build-arg PRE_COMMIT_VERSION=latest \
    --build-arg TERRAFORM_VERSION=latest \
    --build-arg CHECKOV_VERSION=2.0.405 \
    --build-arg INFRACOST_VERSION=latest \
    --build-arg TERRAFORM_DOCS_VERSION=0.15.0 \
    --build-arg TERRAGRUNT_VERSION=latest \
    --build-arg TERRASCAN_VERSION=1.10.0 \
    --build-arg TFLINT_VERSION=0.31.0 \
    --build-arg TFSEC_VERSION=latest \
    .

To disable the pre-commit color output, set -e PRE_COMMIT_COLOR=never.

MacOS

coreutils required for terraform_validate hook on macOS (due to use of realpath).

brew install pre-commit terraform-docs tflint tfsec coreutils checkov terrascan infracost jq
terrascan init
infracost register

Ubuntu 18.04

sudo apt update
sudo apt install -y unzip software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt install -y python3.7 python3-pip
python3 -m pip install --upgrade pip
pip3 install --no-cache-dir pre-commit
python3.7 -m pip install -U checkov
curl -L "$(curl -s https://api.github.com/repos/terraform-docs/terraform-docs/releases/latest | grep -o -E -m 1 "https://.+?-linux-amd64.tar.gz")" > terraform-docs.tgz && tar -xzf terraform-docs.tgz && rm terraform-docs.tgz && chmod +x terraform-docs && sudo mv terraform-docs /usr/bin/
curl -L "$(curl -s https://api.github.com/repos/terraform-linters/tflint/releases/latest | grep -o -E -m 1 "https://.+?_linux_amd64.zip")" > tflint.zip && unzip tflint.zip && rm tflint.zip && sudo mv tflint /usr/bin/
curl -L "$(curl -s https://api.github.com/repos/aquasecurity/tfsec/releases/latest | grep -o -E -m 1 "https://.+?tfsec-linux-amd64")" > tfsec && chmod +x tfsec && sudo mv tfsec /usr/bin/
curl -L "$(curl -s https://api.github.com/repos/accurics/terrascan/releases/latest | grep -o -E -m 1 "https://.+?_Linux_x86_64.tar.gz")" > terrascan.tar.gz && tar -xzf terrascan.tar.gz terrascan && rm terrascan.tar.gz && sudo mv terrascan /usr/bin/ && terrascan init
sudo apt install -y jq && \
curl -L "$(curl -s https://api.github.com/repos/infracost/infracost/releases/latest | grep -o -E -m 1 "https://.+?-linux-amd64.tar.gz")" > infracost.tgz && tar -xzf infracost.tgz && rm infracost.tgz && sudo mv infracost-linux-amd64 /usr/bin/infracost && infracost register

Ubuntu 20.04

sudo apt update
sudo apt install -y unzip software-properties-common python3 python3-pip
python3 -m pip install --upgrade pip
pip3 install --no-cache-dir pre-commit
pip3 install --no-cache-dir checkov
curl -L "$(curl -s https://api.github.com/repos/terraform-docs/terraform-docs/releases/latest | grep -o -E -m 1 "https://.+?-linux-amd64.tar.gz")" > terraform-docs.tgz && tar -xzf terraform-docs.tgz terraform-docs && rm terraform-docs.tgz && chmod +x terraform-docs && sudo mv terraform-docs /usr/bin/
curl -L "$(curl -s https://api.github.com/repos/accurics/terrascan/releases/latest | grep -o -E -m 1 "https://.+?_Linux_x86_64.tar.gz")" > terrascan.tar.gz && tar -xzf terrascan.tar.gz terrascan && rm terrascan.tar.gz && sudo mv terrascan /usr/bin/ && terrascan init
curl -L "$(curl -s https://api.github.com/repos/terraform-linters/tflint/releases/latest | grep -o -E -m 1 "https://.+?_linux_amd64.zip")" > tflint.zip && unzip tflint.zip && rm tflint.zip && sudo mv tflint /usr/bin/
curl -L "$(curl -s https://api.github.com/repos/aquasecurity/tfsec/releases/latest | grep -o -E -m 1 "https://.+?tfsec-linux-amd64")" > tfsec && chmod +x tfsec && sudo mv tfsec /usr/bin/
sudo apt install -y jq && \
curl -L "$(curl -s https://api.github.com/repos/infracost/infracost/releases/latest | grep -o -E -m 1 "https://.+?-linux-amd64.tar.gz")" > infracost.tgz && tar -xzf infracost.tgz && rm infracost.tgz && sudo mv infracost-linux-amd64 /usr/bin/infracost && infracost register

2. Install the pre-commit hook globally

Note: not needed if you use the Docker image

DIR=~/.git-template
git config --global init.templateDir ${DIR}
pre-commit init-templatedir -t pre-commit ${DIR}

3. Add configs and hooks

Step into the repository you want to have the pre-commit hooks installed and run:

git init
cat <<EOF > .pre-commit-config.yaml
repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
  rev: <VERSION> # Get the latest from: https://github.com/antonbabenko/pre-commit-terraform/releases
  hooks:
    - id: terraform_fmt
    - id: terraform_docs
EOF

4. Run

After the pre-commit hook has been installing you can run it manually on all files in the repository.

Local installation:

pre-commit run -a

Docker:

docker run -v $(pwd):/lint -w /lint pre-commit run -a

You be able to list tools versions when needed
TAG=latest && docker run --entrypoint cat pre-commit:$TAG /usr/bin/tools_versions_info

Available Hooks

There are several pre-commit hooks to keep Terraform configurations (both *.tf and *.tfvars) and Terragrunt configurations (*.hcl) in a good shape:

Hook name	Description	Dependencies ^{Install instructions here}
`checkov`	checkov static analysis of terraform templates to spot potential security issues. Hook notes	`checkov` Ubuntu deps: `python3`, `python3-pip`
`infracost_breakdown`	Check how much your infra costs with infracost. Hook notes	`infracost`, `jq`, Infracost API key, Internet connection
`terraform_docs_replace`	Runs `terraform-docs` and pipes the output directly to README.md	`python3`, `terraform-docs`
`terraform_docs_without_` `aggregate_type_defaults`	Inserts input and output documentation into `README.md` without aggregate type defaults. Hook notes same as for terraform_docs	`terraform-docs`
`terraform_docs`	Inserts input and output documentation into `README.md`. Recommended. Hook notes	`terraform-docs`
`terraform_fmt`	Rewrites all Terraform configuration files to a canonical format. Hook notes	-
`terraform_providers_lock`	Updates provider signatures in dependency lock files. Hook notes	-
`terraform_tflint`	Validates all Terraform configuration files with TFLint. Available TFLint rules. Hook notes.	`tflint`
`terraform_tfsec`	TFSec static analysis of terraform templates to spot potential security issues. Hook notes	`tfsec`
`terraform_validate`	Validates all Terraform configuration files. Hook notes	-
`terragrunt_fmt`	Rewrites all Terragrunt configuration files (`*.hcl`) to a canonical format.	`terragrunt`
`terragrunt_validate`	Validates all Terragrunt configuration files (`*.hcl`)	`terragrunt`
`terrascan`	terrascan Detect compliance and security violations.	`terrascan`

Check the source file to know arguments used for each hook.

Hooks usage notes and examples

checkov

For checkov you need to specify each argument separately:

- id: checkov
  args: [
    "-d", ".",
    "--skip-check", "CKV2_AWS_8",
  ]

infracost_breakdown

infracost_breakdown build on top of the infracost breakdown command. It, if needed, runs terraform init, terraform plan and calls infracost API - so this hook can run up to several minutes.

Unlike most other hooks, this one triggers all changes to the files but checks predefined paths each time.

For example, the hook tracks --path=./env/dev and ./env/dev depend on ./main.tf. So when you will make changes to ./main.tf - the hook will run and show the cost changes for ./env/dev.

infracost_breakdown supports custom arguments so you can pass supported flags.
The following example only shows costs:

- id: infracost_breakdown
  args:
    - --args=--path=./env/dev
  verbose: true # Always show costs

Output

Running in "env/dev"

Summary: {
"unsupportedResourceCounts": {
    "aws_sns_topic_subscription": 1
  }
}

Total Monthly Cost:        86.83 USD
Total Monthly Cost (diff): 86.83 USD

You can provide limitations when the hook should fail:
```
- id: infracost_breakdown
  args:
    - --args=--path=./env/dev
    - --hook-config=.totalHourlyCost|tonumber > 0.1
    - --hook-config=.totalHourlyCost|tonumber > 1
    - --hook-config=.projects[].diff.totalMonthlyCost|tonumber != 10000
    - --hook-config=.currency == "USD"
```
Output
```
Running in "env/dev"
Passed: .totalHourlyCost|tonumber > 0.1         0.11894520547945205 >  0.1
Failed: .totalHourlyCost|tonumber > 1           0.11894520547945205 >  1
Passed: .projects[].diff.totalMonthlyCost|tonumber !=10000              86.83 != 10000
Passed: .currency == "USD"              "USD" == "USD"

Summary: {
"unsupportedResourceCounts": {
"aws_sns_topic_subscription": 1
  }
}

Total Monthly Cost:        86.83 USD
Total Monthly Cost (diff): 86.83 USD
```
- Hook uses jq to parse infracost output, so paths to values like .totalHourlyCost and .totalMonthlyCost should be in jq-compatible format.
  To check available structure use infracost breakdown -p PATH_TO_TF_DIR --format json | jq -r . > infracost.json. And play with it on jqplay.org
- Supported comparison operators: <, <=, ==, !=, >=, >.
- Most useful paths and checks:
  - .totalHourlyCost (same to .projects[].breakdown.totalHourlyCost) - show total hourly infra cost
  - .totalMonthlyCost (same to .projects[].breakdown.totalMonthlyCost) - show total monthly infra cost
  - .projects[].diff.totalHourlyCost - show hourly cost diff between existing infra and tf plan
  - .projects[].diff.totalMonthlyCost - show monthly cost diff between existing infra and tf plan
  - .diffTotalHourlyCost (for Infracost version 0.9.12 or newer) or [.projects[].diff.totalMonthlyCost | select (.!=null) | tonumber] | add > 1000 (for Infracost older than 0.9.12):
    - fail if changes push the total monthly cost estimate above $1K
    - fail if changes increase the cost by $1K.
- You can set up only one path per one hook (- id: infracost_breakdown) - this is an infracost limitation.
- Set verbose: true to see cost even when the checks are passed.
- To disable hook color output, set PRE_COMMIT_COLOR=never env var
Docker usage. In docker build or docker run command:
- You need to provide Infracost API key via -e INFRACOST_API_KEY=<your token>. By default it is saved in ~/.config/infracost/credentials.yml
- Set -e INFRACOST_SKIP_UPDATE_CHECK=true to skip the Infracost update check; can be useful in CI/CD systems. Doc

terraform_docs

terraform_docs and terraform_docs_without_aggregate_type_defaults will insert/update documentation generated by terraform-docs framed by markers:
```



```
if they are present in README.md.
It is possible to pass additional arguments to shell scripts when using terraform_docs and terraform_docs_without_aggregate_type_defaults. Send pull-request with the new hook if something is missing.

It is possible to automatically:

create docfile (and PATH to it)
extend existing doc files by appending markers to the end of the file (see item 1)
use different than README.md docfile name.

- id: terraform_docs
  args:
    - --hook-config=--path-to-file=README.md        # Valid UNIX path. I.e. ../TFDOC.md or docs/README.md etc.
    - --hook-config=--add-to-exiting-file=true      # Boolean. true or false
    - --hook-config=--create-file-if-not-exist=true # Boolean. true or false

You can provide arguments to terraform_doc. Eg. for configuration:

- id: terraform_docs
  args:
    - --args=--config=.terraform-docs.yml

If you need some exotic settings, it can be done too. I.e. this one generates HCL files:

- id: terraform_docs
args:
    - tfvars hcl --output-file terraform.tfvars.model .

terraform_docs_replace

terraform_docs_replace replaces the entire README.md rather than doing string replacement between markers. Put your additional documentation at the top of your main.tf for it to be pulled in. The optional --dest argument lets you change the filename that gets created/modified.

Example:

- id: terraform_docs_replace
  args:
    - --sort-by-required
    - --dest=TEST.md

terraform_fmt

terraform_fmt supports custom arguments so you can pass supported flags. Eg:

 - id: terraform_fmt
   args:
     - --args=-no-color
     - --args=-diff
     - --args=-write=false

terraform_providers_lock

The hook requires Terraform 0.14 or later.
The hook invokes two operations that can be really slow:
- terraform init (in case .terraform directory is not initialised)
- terraform providers lock.
Both operations require downloading data from remote Terraform registries, and not all of that downloaded data or meta-data is currently being cached by Terraform.

terraform_providers_lock supports custom arguments:

 - id: terraform_providers_lock
   args:
      - '--args=-platform=windows_amd64'
      - '--args=-platform=darwin_amd64'

It may happen that Terraform working directory (.terraform) already exists but not in the best condition (eg, not initialized modules, wrong version of Terraform, etc.). To solve this problem, you can find and delete all .terraform directories in your repository:
```
echo "
function rm_terraform {
    find . -name ".terraform*" -print0 | xargs -0 rm -r
}
" >>~/.bashrc

# Reload shell and use `rm_terraform` command in the repo root
```
terraform_providers_lock hook will try to reinitialize them before running the terraform providers lock command.

terraform_tflint

terraform_tflint supports custom arguments so you can enable module inspection, deep check mode, etc.

Example:

- id: terraform_tflint
  args:
    - --args=--deep
    - --args=--enable-rule=terraform_documented_variables

When you have multiple directories and want to run tflint in all of them and share a single config file, it is impractical to hard-code the path to the .tflint.hcl file. The solution is to use the __GIT_WORKING_DIR__ placeholder which will be replaced by terraform_tflint hooks with Git working directory (repo root) at run time. For example:
```
- id: terraform_tflint
  args:
    - --args=--config=__GIT_WORKING_DIR__/.tflint.hcl
```

terraform_tfsec

terraform_tfsec will consume modified files that pre-commit passes to it, so you can perform whitelisting of directories or files to run against via files pre-commit flag

Example:
```
- id: terraform_tfsec
  files: ^prd-infra/
```
The above will tell pre-commit to pass down files from the prd-infra/ folder only such that the underlying tfsec tool can run against changed files in this directory, ignoring any other folders at the root level

To ignore specific warnings, follow the convention from the documentation.

Example:

resource "aws_security_group_rule" "my-rule" {
    type = "ingress"
    cidr_blocks = ["0.0.0.0/0"] #tfsec:ignore:AWS006
}

terraform_tfsec supports custom arguments so you can pass supported --no-color or --format (output), -e (exclude checks) flags:

 - id: terraform_tfsec
   args:
     - >
       --args=--format json
       --no-color
       -e aws-s3-enable-bucket-logging,aws-s3-specify-public-access-block

When you have multiple directories and want to run tfsec in all of them and share a single config file - use the __GIT_WORKING_DIR__ placeholder. It will be replaced by terraform_tfsec hooks with Git working directory (repo root) at run time. For example:
```
- id: terraform_tfsec
  args:
    - --args=--config-file=__GIT_WORKING_DIR__/.tfsec.json
```
Otherwise, will be used files that located in sub-folders:
```
- id: terraform_tfsec
  args:
    - --args=--config-file=.tfsec.json
```

terraform_validate

terraform_validate supports custom arguments so you can pass supported -no-color or -json flags:

 - id: terraform_validate
   args:
     - --args=-json
     - --args=-no-color

terraform_validate also supports custom environment variables passed to the pre-commit runtime:

- id: terraform_validate
  args:
    - --envs=AWS_DEFAULT_REGION="us-west-2"
    - --envs=AWS_ACCESS_KEY_ID="anaccesskey"
    - --envs=AWS_SECRET_ACCESS_KEY="asecretkey"

It may happen that Terraform working directory (.terraform) already exists but not in the best condition (eg, not initialized modules, wrong version of Terraform, etc.). To solve this problem, you can find and delete all .terraform directories in your repository:
```
echo "
function rm_terraform {
    find . -name ".terraform*" -print0 | xargs -0 rm -r
}
" >>~/.bashrc

# Reload shell and use `rm_terraform` command in the repo root
```
terraform_validate hook will try to reinitialize them before running the terraform validate command.

Warning: If you use Terraform workspaces, DO NOT use this workaround (details). Wait to force-init option implementation

Authors

This repository is managed by Anton Babenko with help from these awesome contributors:

License

MIT licensed. See LICENSE for full details.

26 KiB Raw Blame History

Collection of git hooks for Terraform to be used with pre-commit framework

How to install

1. Install dependencies

2. Install the pre-commit hook globally

3. Add configs and hooks

4. Run

Available Hooks

Hooks usage notes and examples

checkov

infracost_breakdown

terraform_docs

terraform_docs_replace

terraform_fmt

terraform_providers_lock

terraform_tflint

terraform_tfsec

terraform_validate

Authors

License

26 KiB

Raw Blame History