Unix convert windows file to unix

You may ask yourself, what is the purpose of this article? Why convert a Windows file to adapt to a UNIX environment like Linux? Isn’t Linux all-powerful? The exceptional capabilities of the Linux operating system do not spare it from incompatible displays of files transferred from other computing platforms.

Just because you can open a file on a Linux environment does not imply that you have full control over how the file’s texts should be displayed.

[ You might also like: How to Find Files Containing Specific Text String in Linux ]

You will encounter instances where a file’s texts or words are jammed together on a single giant line. In other instances, the same file texts’ displays might not have line breaks or sentence spacing.

A common attribute of raw Windows files opened in UNIX systems like Linux is the unavoidable end-of-line display of ^M or Ctrl-M characters.

This article guide seeks to achieve one objective; the conversion of a Windows File to a UNIX file without changing the format of the resulting file.

Ways to Convert Windows File to Unix Format in Linux

We can achieve the objective of our article through several methods. These methods allow us to convert a Windows file to a UNIX file and still retain the original format of the Windows file.

Convert Windows File to Unix Using dos2unix Command

Depending on your Linux operating system distribution, you can install the dos2unix command-line tool from one of the following commands:

$ sudo apt-get install dos2unix     [On Debian, Ubuntu and Mint]
$ sudo yum install dos2unix         [On RHEL/CentOS/Fedora and Rocky Linux/AlmaLinux]
$ sudo emerge -a sys-apps/dos2unix  [On Gentoo Linux]
$ sudo pacman -S dos2unix           [On Arch Linux]
$ sudo zypper install dos2unix      [On OpenSUSE]    

The command syntax for using the dos2unix tool is as follows:

$ dos2unix Your_Windows_File Final_Unix_File

So if you have a sample file created on a Windows computing system and want to open it on a Linux computing system without compromising its format, you would use the following command.

$ dos2unix windows_readme.txt unix_readme.txt

Before we run the above command, we need to create a blank unix_readme.txt file that will accommodate the converted file.

$ touch unix_readme.txt
$ dos2unix windows_readme.txt unix_readme.txt    

Картинка с сайта: www.ubuntumint.com

As per the screen capture, your converted Windows file should comfortably adapt to any Unix environment.

Using AWK Command – Convert Windows File to Unix

The awk command is pre-installed on all modern full-fledged UNIX computing systems like Linux. To convert our sample Windows file, we would implement the awk command in the following manner:

$ awk '{ sub("\r$", ""); print }' windows_readme.txt > new_unix_readme.txt

Картинка с сайта: www.ubuntumint.com

As you have noted, with the awk command, we don’t need a pre-existing blank Linux file to accommodate the converted Windows file. The command creates and populates the Unix file version of the Windows file on its own.

Using tr Command – Convert Windows File to Unix

If most of the Windows files you open on your Linux environment have unnecessary Ctrl-Z and carriage return characters, then you will appreciate what the tr command has to offer.

Supposing our sample Windows file is a victim of such characters, removing them will require implementing the following command:

$ tr -d '' < windows_readme.txt > polished_unix_readme.txt

Картинка с сайта: www.ubuntumint.com

The inbuilt nature of the tr command also generates the resulting UNIX file without the need for its pre-creation.

The flexibility of the three discussed approaches to converting any editable Windows file to UNIX file format should save you from the headaches of having to manually edit your downloaded or transferred Windows files to remove unwanted characters and spaces while on a Linux environment.

When working with files across different operating systems, text file format compatibility becomes crucial. One of the most common issues that Linux users encounter involves line ending differences. Files created on Windows systems use different line endings than those created on Linux, which can cause scripts to fail, configuration files to malfunction, and text to display incorrectly. The dos2unix utility offers a simple and effective solution to this problem.

Introduction to Dos2unix

The dos2unix command is a powerful utility in Linux that converts text files from DOS/Windows format to Unix format. This conversion is essential because different operating systems use different character combinations to represent line endings in text files. DOS and Windows use a combination of two characters—Carriage Return (CR) followed by Line Feed (LF)—to mark the end of a line. This combination is often represented as CRLF or \r\n. Unix and Linux, on the other hand, use only a Line Feed (LF or \n) character.

When files are transferred between operating systems, these line ending differences can cause compatibility issues. Scripts might fail to execute properly, configuration files could be misinterpreted, and text editors might display extra characters or incorrect formatting. The dos2unix utility elegantly solves these problems by converting DOS/Windows line endings to Unix format.

The utility is part of a package that also includes unix2dos, which performs the reverse conversion from Unix to DOS format. Together, these tools ensure seamless file compatibility across different operating systems.

Understanding Line Endings

Before diving deeper into dos2unix usage, it’s important to understand the different line ending formats used across various operating systems:

DOS/Windows Format (CRLF: \r\n): Windows-based systems use a two-character sequence—Carriage Return followed by Line Feed (CRLF, or \r\n)—to indicate the end of a line. This tradition dates back to the days of typewriters where a carriage return would move the carriage back to the start of the line, and a line feed would advance the paper to the next line.

Unix/Linux Format (LF: \n): Unix-based systems, including Linux and modern macOS, use only a Line Feed (LF, or \n) character to mark line endings. This simpler approach is preferred in the Unix world.

Mac Format (CR: \r): Older Mac systems (pre-OS X) used only Carriage Return (CR, or \r) for line endings. Modern macOS now uses the Unix-style LF format.

When viewing a DOS/Windows text file in a Unix environment, the extra Carriage Return character might appear as a ^M symbol at the end of each line. This special character can cause various issues:

• Shell scripts may fail to execute
• Configuration files might be misinterpreted
• Applications could malfunction or produce unexpected results
• Text may display with visible control characters
• File processing tools might handle the file incorrectly

Understanding these differences is crucial when working in mixed environments or when transferring files between different operating systems.

Installing Dos2unix

The dos2unix utility is not installed by default on most Linux distributions, but it’s readily available in standard repositories. Here’s how to install it on various Linux distributions:

For Ubuntu, Debian, and Debian-based distributions:

sudo apt install dos2unix

For CentOS, RHEL, and other RPM-based distributions:

sudo yum install dos2unix

For Fedora:

sudo dnf install dos2unix

For Arch Linux and Arch-based distributions:

sudo pacman -S dos2unix

After installation, you can verify that dos2unix is correctly installed by checking its version:

dos2unix --version

This command should display the version information of the installed dos2unix utility. The version information is useful to keep track of available features, as newer versions might include additional functionalities or bug fixes.

The dos2unix package typically includes both the dos2unix and unix2dos utilities, allowing for bidirectional conversion between DOS and Unix formats.

Basic Syntax and Usage

The fundamental syntax of the dos2unix command is straightforward:

dos2unix [options] [file...]

In this syntax:

• [options] represents various parameters that modify the command’s behavior
• [file...] indicates one or more files to be converted

When used without any options, dos2unix will convert the specified file in place, which means it will overwrite the original file with the converted version. This is known as “old file mode” or “in-place conversion”.

For example, to convert a file named script.sh from DOS to Unix format:

dos2unix script.sh

Upon successful execution, the command typically outputs a message like:

dos2unix: converting file script.sh to Unix format...

To verify that the conversion was successful, you can use commands like cat -v or hexdump to examine the file’s content and confirm that the carriage return characters have been removed.

If you don’t specify any files, dos2unix will read from standard input and write to standard output. This allows you to use dos2unix in command pipelines:

cat dosfile.txt | dos2unix > unixfile.txt

This flexibility makes dos2unix a versatile tool that can be incorporated into various text processing workflows.

Command Options and Flags

The dos2unix command offers numerous options to customize its behavior. Here are the most commonly used options:

-f, –force: Force conversion even if the file is not in DOS format. This can be useful when you want to ensure consistency across multiple files regardless of their current format.

dos2unix -f config.txt

-k, –keepdate: Preserve the original file’s timestamp. By default, dos2unix updates the timestamp to reflect the conversion time, but this option maintains the original timestamp.

dos2unix -k important_document.txt

-q, –quiet: Suppress all warnings and messages during conversion. This is useful for scripting or when processing numerous files.

dos2unix -q *.txt

-n, –newfile: Create a new file instead of overwriting the original. This option requires specifying both input and output filenames.

dos2unix -n original.txt converted.txt

-b, –keep-bom: Keep the Byte Order Mark (BOM) if present. This is important when dealing with UTF-encoded files.

dos2unix -b unicode_file.txt

-r, –recursive: Process files in directories recursively. This is not a native option in all versions of dos2unix, but can be achieved using find and xargs in Unix systems.

-c mode, –convmode mode: Set conversion mode. Available options include ASCII, 7bit, ISO, and MAC. The default is usually ASCII.

dos2unix -c ascii document.txt

A comprehensive table of command options can be found in the man pages, accessible via man dos2unix on your Linux system.

Understanding these options allows for flexible and precise file conversions based on specific requirements, making dos2unix a powerful tool in a Linux user’s toolkit.

Basic Conversion Examples

Let’s explore some basic examples of using dos2unix for file conversion:

Converting a Single File:
The most straightforward use case is converting a single file:

dos2unix myfile.txt

This command converts myfile.txt from DOS format to Unix format, overwriting the original file.

Creating a Before and After Comparison:
To see the differences before and after conversion:

# Before conversion
cat -v dosfile.txt  # May show ^M characters at line endings

# Perform conversion
dos2unix dosfile.txt

# After conversion
cat -v dosfile.txt  # No ^M characters should be visible

Converting Multiple Files at Once:
You can specify multiple files on the command line:

dos2unix file1.txt file2.txt file3.txt

All the specified files will be converted to Unix format.

Using Wildcards for Batch Conversion:
Wildcards allow converting multiple files matching a pattern:

dos2unix *.txt

This converts all .txt files in the current directory.

Preserving Original Files:
To create converted copies while keeping originals:

dos2unix -n original.txt converted.txt

This reads original.txt, converts it to Unix format, and saves the result as converted.txt without modifying the original file.

Converting with Backup Creation:
If you want to convert in place but keep a backup:

dos2unix -b myfile.txt

This creates a backup file named myfile.txt.bak before converting myfile.txt.

These basic examples cover the most common use cases and provide a foundation for understanding how dos2unix operates on files. As you become more familiar with the command, you can explore more advanced options and techniques.

Advanced Conversion Examples

For more complex scenarios, dos2unix offers advanced options and can be combined with other commands:

Converting Files While Preserving Timestamps:
To maintain the original file’s timestamp after conversion:

dos2unix -k important_config.txt

This is useful when file modification times need to be preserved for auditing or version control purposes.

Creating New Files Instead of Overwriting:
For safer operation, especially with critical files:

dos2unix -n source.txt destination.txt

This approach is recommended when you want to ensure the original file remains untouched.

Working with Different Conversion Modes:
For special character handling requirements:

dos2unix -c 7bit international_text.txt

This converts the file while ensuring 7-bit ASCII compatibility.

Recursive Directory Conversion:
To convert all text files in a directory structure:

find . -type f -name "*.txt" -print0 | xargs -0 dos2unix

This command finds all .txt files in the current directory and subdirectories and converts them using dos2unix.

Handling Special Character Encodings:
For files with specific encoding requirements:

dos2unix -ul unicode_file.txt

This flag assumes UTF-16LE encoding for the input file.

Converting Files Across Different Filesystems:
When converting between different filesystems, use:

cd /target_directory
dos2unix -n /source_filesystem/file.txt ./converted_file.txt

This avoids issues with rename operations that might occur between different filesystems.

Multi-Processor Batch Processing:
For large directories with many files:

find . -type f -print0 | xargs -0 -n 1 -P 4 dos2unix

This processes files using 4 processors in parallel, which can significantly speed up conversion of large file sets.

Skip Binary Files and Only Convert Text Files with Windows Line Endings:

find . -type f -print0 | xargs -0 dos2unix -ic0 | xargs -0 dos2unix -b

This command first identifies only files with Windows line breaks and then converts them while preserving UTF-8 BOM if present.

These advanced examples demonstrate the flexibility and power of dos2unix when handling complex file conversion scenarios across different environments and requirements.

Unix2dos: The Reverse Conversion

While dos2unix converts from DOS to Unix format, there are situations where you need to perform the reverse conversion. This is where the unix2dos utility comes in.

Understanding unix2dos:
The unix2dos command converts files from Unix format (LF line endings) to DOS format (CRLF line endings). It’s part of the same package as dos2unix and follows similar syntax and options.

Basic Usage:
The basic syntax for unix2dos is similar to dos2unix:

unix2dos [options] [file...]

To convert a file from Unix to DOS format:

unix2dos unixfile.txt

This converts the file in place, adding carriage return characters before each line feed.

When to Use unix2dos:
Common scenarios for using unix2dos include:

• Preparing files for Windows users
• Creating configuration files for Windows applications
• Ensuring proper text file display in DOS/Windows environments
• Preparing scripts that will run in Windows command prompt

Creating a Backup During Conversion:
Similar to dos2unix, you can create a backup of the original file:

unix2dos -b unixfile.txt

This creates a backup file with a .bak extension before performing the conversion.

Converting to a New File:
To preserve the original Unix file:

unix2dos -n unixfile.txt dosfile.txt

This reads unixfile.txt, converts it to DOS format, and saves it as dosfile.txt.

The unix2dos utility complements dos2unix, providing a complete solution for text file format conversion between operating systems. Together, they ensure seamless file compatibility regardless of the target environment.

Integration with Scripts and Workflows

The dos2unix utility can be effectively integrated into scripts and automated workflows:

Creating a Bash Script for Batch Conversion:
Here’s a simple bash script that converts all text files in a directory:

#!/bin/bash
# Convert all text files in the current directory to Unix format

for file in *.txt; do
    echo "Converting $file..."
    dos2unix "$file"
done

echo "Conversion completed!"

Automating Conversion in Development Workflows:
For development teams working across different platforms, you can create a pre-commit hook in Git:

#!/bin/bash
# Pre-commit hook to ensure all committed text files use Unix line endings

# Get list of staged text files
files=$(git diff --cached --name-only --diff-filter=ACM | grep -E '\.(txt|md|sh|py|java|c|cpp|h|xml|json|yaml|yml)$')

if [ -n "$files" ]; then
    echo "Converting line endings to Unix format..."
    echo "$files" | xargs dos2unix
    echo "$files" | xargs git add
fi

Integrating with CI/CD Pipelines:
In a Continuous Integration pipeline, you might include a step like:

convert-line-endings:
  stage: prepare
  script:
    - find . -type f -name "*.sh" -o -name "*.py" | xargs dos2unix

Processing Uploaded Files Automatically:
For web applications that receive file uploads:

// PHP example for processing uploaded files
function processUpload($tempFile, $targetFile) {
    // Copy uploaded file to target location
    copy($tempFile, $targetFile);
    
    // Convert line endings if it's a text file
    $mimeType = mime_content_type($targetFile);
    if (strpos($mimeType, 'text/') === 0) {
        exec("dos2unix " . escapeshellarg($targetFile));
    }
}

Git Configuration for Line Endings:
Git offers configuration options to handle line endings:

# Set to auto-convert CRLF to LF on commit
git config --global core.autocrlf input

# Alternative: show warnings but don't convert
git config --global core.safecrlf warn

These integration examples show how dos2unix can be incorporated into various workflows to ensure consistent file formats across different environments and development processes.

Alternative Methods for Line Ending Conversion

While dos2unix is a dedicated tool for line ending conversion, Linux offers several alternative methods:

Using sed:
The sed (stream editor) command can remove carriage returns:

sed 's/\r$//' dosfile.txt > unixfile.txt

This regular expression removes carriage returns at the end of each line.

Using tr:
The tr (translate) command can delete specific characters:

tr -d '\r' < dosfile.txt > unixfile.txt

This removes all carriage return characters from the input file.

Using perl One-Liners:
Perl offers powerful text processing capabilities:

perl -pi -e 's/\r\n/\n/g' file.txt

This replaces CRLF with LF in place.

Using Vi/Vim:
The Vim text editor can handle line ending conversion:

1. Open the file: vim file.txt
2. Set the file format: :set fileformat=unix
3. Save the file: :wq

Comparison of Methods:

When to Use Alternatives:

• When dos2unix is not available on the system
• For quick, one-off conversions in scripts
• When integrated with other text processing operations
• For systems with minimal dependencies

These alternative methods provide flexibility when dos2unix isn’t available or when specific requirements make other tools more suitable for the task.

Troubleshooting Common Issues

When using dos2unix, you might encounter various issues. Here’s how to resolve them:

Permission Errors:
If you see “Permission denied” errors:

dos2unix: Failed to open input file: Permission denied

Solution: Check file permissions and ownership:

chmod +rw filename.txt

Or run the command with sudo if appropriate.

Binary File Warnings:
When attempting to convert a binary file:

dos2unix: Binary symbol found at line X

Solution: Skip binary files or use the -f flag to force conversion (not recommended for most binary files):

dos2unix -f filename.bin

Note that forcing conversion of binary files can corrupt them.

File Ownership Issues:
When dos2unix fails to change ownership:

dos2unix: Failed to change the owner and group of temporary output file

Solution: Use the new file mode instead of in-place conversion:

dos2unix -n oldfile.txt newfile.txt
mv -f newfile.txt oldfile.txt

This avoids permission issues related to file ownership preservation.

Problems Converting Between Filesystems:
Errors when source and target are on different filesystems:

dos2unix: problems renaming './tmpfile' to '/mnt/target/file'

Solution: Change to the target directory first:

cd /mnt/target
dos2unix -n /source/file ./file

This avoids issues with the rename operation across filesystems.

Command Not Found Error:
If you encounter:

-bash: dos2unix: command not found

Solution: Install the dos2unix package using your distribution’s package manager:

# Debian/Ubuntu
sudo apt install dos2unix

# CentOS/RHEL
sudo yum install dos2unix

Partial Conversions:
If only some files are converted in a batch:
Solution: Check for errors in the output and process files individually to identify problematic ones.

Addressing these common issues will help ensure smooth and successful file conversions with dos2unix across various environments and scenarios.

Best Practices

To get the most out of dos2unix, follow these recommended practices:

Always Make Backups of Important Files:
Before performing in-place conversions, create backups:

cp important_file.txt important_file.txt.backup

Or use the -b option to have dos2unix create backups automatically.

Document File Format Conversions:
Keep records of which files have been converted, especially for critical system files or configuration files.

Use the Appropriate Conversion Method:

• Use dos2unix for individual files or small batches
• Use find/xargs for recursive directory conversion
• Use git configurations for development projects
• Consider sed/tr for quick one-off conversions in scripts

Check Conversion Success:
After conversion, verify that line endings have been correctly converted:

cat -vET filename.txt

This shows special characters including carriage returns (^M).

Consider File Encoding:
Be aware of file encodings beyond line endings. Use appropriate flags like -u for UTF-16 files or -b to preserve BOMs when necessary.

Handle Line Endings Consistently:
Establish consistent standards for your project:

• Use .gitattributes for Git projects
• Document line ending expectations
• Set up pre-commit hooks to enforce standards

Be Cautious with System Files:
Take extra care when converting system configuration files or scripts, as incorrect conversion could affect system functionality.

Performance Considerations:
For large conversions:

• Process files in batches
• Use multi-processor approaches for large directories
• Consider memory usage and disk I/O impact

Following these best practices will help ensure smooth and reliable file conversions while minimizing risks and potential issues.

A Quick Intro to End-Of-Line

Most people don’t realise that when they hit the Enter key to create a new paragraph in a text file, something very different is going on behind the scenes in the three major operating systems: Windows, Macintosh and Linux. The “end-of-line delimiter” (often expressed as “End-Of-Line“, “End of Line“, or just “EOL“) – which some of you know as the “line break” or “newline” – is a special character used to designate the end of a line within a text file.

UNIX-based operating systems (like all Linux distros and BSD derivatives) use the line feed character (\n or <LF>), “classic” Mac OS uses a carriage return (\r or <CR>), while DOS/Windows uses a carriage return followed by a line feed (\r\n or <CR><LF>). Now that Mac OS X is based on FreeBSD‘s file system, it follows the UNIX convention.

Now, the reason most people don’t know about all this is because nobody really should have to. But while users of Linux distros and Mac OS can open Windows text files in basically any available editor and not even know the difference, the same can’t be said for Windows users opening files created in one of the other operating systems.

If you type up a simple text file in Ubuntu and save it in the default “Unix/Linux” format, in Windows it will appear as one continuous paragraph, with black squares where the line breaks (or new paragraphs) should be. While you can open the file in a more advanced text editor (or proper word processor) to view it as it should look, others you’ve sent it to are just likely to double-click it and let it open in Notepad (which can only handle MS-DOS EOL).

Occasionally, the reverse is the issue, but you can convert Windows text files to UNIX easily with Gedit, as well as convert them via the terminal, so hopefully the following guide will be of use.

For more detailed info on End-Of-Line, go to the Wikipedia page.

Or if you’re wanting to do the reverse, check out how to convert to Windows format via the terminal and with Save As… in Gedit.

Converting Windows EOL to Linux via the Terminal

If you find the text editor you’re using to display Windows files in Ubuntu shows ^M instead of a line break (not very likely with even the most lightweight text editors, but something you’ll probably come across if you display the text in a terminal), don’t worry – just convert them to Unix/Linux format.

While you can actually open them in Gedit and use Save As… to save over them (or to create copies) in the correct format, for more than a couple of files this would be the long, complicated solution.

By far the quickest and easiest approach is to convert the offending files via the command-line. This way, you could batch-convert hundreds of such files at once, not have to do them individually.
There are actually quite a few ways to do this, but we’ll look at a couple of tiny packages you can install, and the related commands to use.

The first – the tofrodos package – is undoubtedly the most widely-used, so we’ll look at that in detail – especially since many of the guides out there are outdated, since the commands it contains have been renamed.

The second is a little package called flip, and since it’s tiny and won’t cause any issues, it’s worth installing as a backup (just in case. I found it useful after trying to get tofrodos going on a new system, before I found out the commands were changed).

There is no actual command tofrodos, as it is just the package that contains the commands todos and fromdos. Currently, the vast majority of online guides will list the commands as unix2dos and dos2unix, but as the developer states:

With this release the symlinks “unix2dos” and “dos2unix” are dropped from the package. This will allow the introduction of the original dos2unix package, which also supports conversion to MacOS style files.

So now you can choose to use either todos (to convert to Windows) and fromdos (to convert to Linux), or just fromdos with options (fromdos -u to convert to DOS, and fromdos -d to convert to UNIX, though obviously the -d option really isn’t needed, as it is the default behaviour for the fromdos command).

We’ll use fromdos, as it is easier to remember, and show how to alter a single file, or all text files in a given folder. When you’re ready to proceed, open a terminal in the folder containing the text file(s) and use one of the following commands (note that for the purpose of illustration, the .txt suffix is used, but you can specify any other extension for your text files).

To Convert to UNIX/Linux via Terminal:

Single file (remember to replace filename.txt with the actual name of the file)

fromdos filename.txt

All text files in a folder (if the extension differs to .txt, simply replace it in the command)

fromdos *.txt

Similarly, flip is easy to use:

flip -u filename.txt (or flip -u *.txt for multiple files)

Converting Windows EOL to Linux with Gedit

It’s actually very easy to convert text files with Windows EOL to Unix/Linux in Ubuntu using the default Text Editor, Gedit. Simply open the files, choose Save As…, go to Line Ending in the dialogue box and choose Unix/Linux instead of Windows. While that is easy enough, for more than one or two you’d really want to save yourself some time and hassle and perform a batch-conversion via the terminal.

Картинка с сайта: ubuntugenius.wordpress.com

☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻☻

Did this information make your day? Did it rescue you from hours of headache? Then please consider making a donation via PayPal, to buy me a donut, beer, or some fish’n’chips for my time and effort! Many thanks!

Was this article helpful?

YesNo