The Golang dependency tool dep is awesome, but it has this habit of marking files as changed when the only difference is related to line-endings. This blog post highlights the issue and provides a solution.

The Problem

When dep updates a project’s dependencies, files are fetched from remote version control systems (VCS) such as Git, Mercurial, etc. However, there’s this slight problem with the process. Sometimes dozens, if not hundreds, of files are marked as modified because of their line-endings. For example, the following command lists all files currently marked as modified:

$ git status
On branch feature/docker-csi-bridge
Your branch is up-to-date with origin/feature/docker-csi-bridge'.
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)
modified: Gopkg.lock
modified: agent/csi/mod_csi_docker.go
modified: vendor/github.com/Microsoft/go-winio/ea.go
modified: vendor/github.com/Microsoft/go-winio/ea_test.go
modified: vendor/github.com/Microsoft/go-winio/vhd/mksyscall_windows.go
modified: vendor/github.com/Microsoft/go-winio/vhd/vhd.go
modified: vendor/github.com/hashicorp/hcl/.gitignore
modified: vendor/github.com/hashicorp/hcl/Makefile
modified: vendor/github.com/hashicorp/hcl/hcl/parser/test-fixtures/assign_deep.hcl
modified: vendor/github.com/hashicorp/hcl/hcl/parser/test-fixtures/comment_crlf.hcl
modified: vendor/github.com/hashicorp/hcl/hcl/parser/test-fixtures/complex_crlf.hcl
modified: vendor/github.com/hashicorp/hcl/hcl/parser/test-fixtures/list.hcl
modified: vendor/github.com/hashicorp/hcl/hcl/parser/test-fixtures/multiple.hcl
modified: vendor/github.com/hashicorp/hcl/hcl/parser/test-fixtures/structure_basic.hcl
modified: vendor/github.com/hashicorp/hcl/hcl/parser/test-fixtures/structure_empty.hcl
modified: vendor/github.com/hashicorp/hcl/hcl/parser/test-fixtures/types.hcl
modified: vendor/github.com/hashicorp/hcl/hcl/printer/testdata/comment_crlf.input
modified: vendor/github.com/hashicorp/hcl/hcl/test-fixtures/comment.hcl
modified: vendor/github.com/hashicorp/hcl/hcl/test-fixtures/list.hcl
modified: vendor/github.com/hashicorp/hcl/hcl/test-fixtures/multiple.hcl
modified: vendor/github.com/hashicorp/hcl/hcl/test-fixtures/structure_basic.hcl
modified: vendor/github.com/hashicorp/hcl/hcl/test-fixtures/structure_empty.hcl
modified: vendor/github.com/hashicorp/hcl/hcl/test-fixtures/types.hcl
modified: vendor/github.com/hashicorp/hcl/json/parser/test-fixtures/array.json
modified: vendor/github.com/hashicorp/hcl/json/parser/test-fixtures/basic.json
modified: vendor/github.com/hashicorp/hcl/json/parser/test-fixtures/object.json
modified: vendor/github.com/hashicorp/hcl/json/parser/test-fixtures/types.json
modified: vendor/github.com/hashicorp/hcl/json/test-fixtures/array.json
modified: vendor/github.com/hashicorp/hcl/json/test-fixtures/basic.json
modified: vendor/github.com/hashicorp/hcl/json/test-fixtures/object.json
modified: vendor/github.com/hashicorp/hcl/json/test-fixtures/types.json
modified: vendor/github.com/hashicorp/hcl/test-fixtures/assign_deep.hcl
modified: vendor/github.com/hashicorp/hcl/test-fixtures/basic.hcl
modified: vendor/github.com/hashicorp/hcl/test-fixtures/basic.json
modified: vendor/github.com/hashicorp/hcl/test-fixtures/flat.hcl
modified: vendor/github.com/hashicorp/hcl/test-fixtures/structure.json
modified: vendor/github.com/hashicorp/hcl/test-fixtures/structure2.hcl
modified: vendor/github.com/hashicorp/hcl/test-fixtures/structure2.json
modified: vendor/github.com/hashicorp/hcl/test-fixtures/structure_flat.json
modified: vendor/github.com/hashicorp/hcl/test-fixtures/structure_flatmap.hcl
modified: vendor/github.com/onsi/gomega/matchers/test_data/xml/sample_05.xml
modified: vendor/github.com/pelletier/go-toml/example-crlf.toml
modified: vendor/github.com/sirupsen/logrus/appveyor.yml
modified: vendor/github.com/spf13/afero/.travis.yml

In the above example only the first two files should be marked as modified:

$ git diff agent/csi/mod_csi_docker.go
diff --git a/agent/csi/mod_csi_docker.go b/agent/csi/mod_csi_docker.go
index 2a1adea2..d4d82092 100644
--- a/agent/csi/mod_csi_docker.go
+++ b/agent/csi/mod_csi_docker.go

[0][email protected]:rexray$ git diff Gopkg.lock
diff --git a/Gopkg.lock b/Gopkg.lock
index f5115745..cd200ed2 100644
--- a/Gopkg.lock
+++ b/Gopkg.lock

The remaining files have been incorrectly updated as a result of dep ensure:

$ git diff vendor/github.com/hashicorp/hcl/json/parser/test-fixtures/basic.json
warning: CRLF will be replaced by LF in vendor/github.com/hashicorp/hcl/json/parser/test-fixtures/basic.json.
The file will have its original line endings in your working directory.

$ git diff vendor/github.com/hashicorp/hcl/hcl/parser/test-fixtures/multiple.hcl
warning: CRLF will be replaced by LF in vendor/github.com/hashicorp/hcl/hcl/parser/test-fixtures/multiple.hcl.
The file will have its original line endings in your working directory.

The Solution

The following one-liner is specific to Git, and it checks out all files marked as modified due only to their line-endings:

git checkout -- \
  $(for f in $(git status --porcelain | \
    grep " M" | awk '{print $2}'); do \
      if git --no-pager diff $f 2>&1 | head -n 1 | grep -q "warning: CRLF will be replaced by LF"; then \
        echo $f; \
      fi; \
    done;)

Run the above script and then use git status one more time to show that all modified files affected by line-ending changes have been reverted, and files affected by other changes were ignored:

$ git status
On branch feature/docker-csi-bridge
Your branch is up-to-date with 'origin/feature/docker-csi-bridge'.
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)
modified: Gopkg.lock
modified: agent/csi/mod_csi_docker.go

  • pdffs

    This doesn’t seem like a thing dep would do. Are you sure your git config is not set to change line endings?