The eflow blog

I’ve spoken previously about the travails of having a developer community that uses both Windows and Unix-based machines: merging files edited in both environments. Hopefully those days are over. (At least for the OpenMRS community)

Chase Yarbrough attempted to fix this by adding the svn:eol-style to all of the files in the OpenMRS repository recursively. The problem is that all files that were already committed from both operating systems failed with:

svn: File ‘somefilename.ext’ has inconsistent newlines
svn: Inconsistent line ending style

However, merging continued to be a pain with those files that were skipped over.

Solution

I created a short script to recursively loop over all files in the root directory, run dos2unix, and then set the svn:eol-style to CRLF:

Download this script

#!/bin/sh

# This script normalizes the newlines in all files in
# a directory and adds the svn:eol-style property
# Author: Ben Wolfe - 10/2008
# http://blog.eflow.org

path=`pwd`

# get a flat list of all files under the current directory
f=`ls -RlQ | awk -v PATH=$path '{ \
if ($1 ~ /.*:$/) \
path = substr($1,3,length($1)-4); \
else \
if ($0 ~ /^-/) \
printf("%s%s/%s\n", PATH, path, \
substr($0, match($0,"\".*\"")+1, RLENGTH-2) \
) \
}'`

for i in $f
do
#ignore the build folder contents
if ! echo $i | grep build > /dev/null;
then
        #check subversion for this property
        x=`svn propget svn:eol-style $i`
        if [ ! -n "$x" ]
        then
                # get the file's extension
                ext=`echo "$i" | awk -F . '{print $NF}'`

                # This is an "if list contains" method for shell scripts
                # check if ext is one of the text based extensions
                case $ext in
                        java|jsp|css|js|xml|sql|txt|properties|tld|tag|html|htm|sh)
                                #echo No svn:eol-style prop: $i . Adding now.
                                dos2unix -a $i
                                svn propset svn:eol-style "CRLF" $i
                                ;;
                        *)
                                echo invalid extension: $ext
                                ;;
                esac
        fi
fi
done

echo "done"

I committed all changes from running this script in changeset 5893.

This is in the openmrs category tagged as scripting, subversion, svn

7 comments »

7 Responses

1. Burke Says:
   November 3rd, 2008 at 11:56 am

   Do all source files committed to the repository have to have this property set — i.e., will we have to do this repeatedly over time as new files are added to the repository without the svn:eol-style property set? If so, is there an alternative (permanent) solution — e.g., automatically apply this property to all source files via an svn hook?

2. Ben Says:
   November 3rd, 2008 at 12:00 pm

   That is an excellent point, Burke. And I just forgot to mention my solution for that. I added the svn:default-eol-style property to the root folder so that all new files get the CRLF property by default. See http://dev.openmrs.org/browser/openmrs/trunk?rev=5893

3. The eflow blog » Merge Problems with Inconsistent Newlines in SVN Says:
   December 18th, 2008 at 2:32 pm

   [...] the fix_newlines.sh [...]

4. chase Says:
   June 8th, 2009 at 8:45 pm

   Why did we end up doing dos2unix and then CRLF…. that seems wrong. Shouldn’t it have been unix2dos and then CRLF or dos2unix and then LF??

5. Ben Says:
   June 9th, 2009 at 8:56 am

   I used dos2unix because I just needed a standard line ending before applying CRLF. Yes, it probably would have made more sense to use unix2dos…if I had known it existed.

6. Adam Bruss Says:
   May 16th, 2011 at 2:21 pm

   Thank you very much this worked for me.

7. Luke Shumaker Says:
   August 9th, 2012 at 9:39 am

   It would have been simpler to use `find` to get a list of files. `find “$path” -type f`

Leave a Comment