How to remove ^M and other non-printable characters from the file
If you are working or playing around with files on different O.S., and you are just copying/moving files from one O.S. based system to other. Unknowingly you are inviting non-printable character to come with! This specially happens when you open any file in Linux using any editor (vim in my case), which you got from windows based machines. You will see ^M at the end of each line (It is usually treated as carriage return/linefeed in Linux environment which we got from its step brother Windows :p ).
What is this ^M and how it came into the picture?
^M is Windows carriage return: Unix uses for newline 0xA, Windows a combination of two characters: 0xA(10 in decimal) 0xD(13 in decimal), ^M happens to be 0xD. It’s ASCII value of \n and \r respectively.
Usually whatever starts with ^ is treated as non-printable character (but don’t be so sure!). Like ^L or ^@ etc.How I will find if it’s printable or non-printable character?
Just try to open a file using cat without option and cat with option –v and observe its outputs.
Ex: cat test.txt and cat –v test.txt (-v options prints non-printable character too.)
Output using cat command without –v cat test.txt
||This is first line
|| Lets see if ^M is rintable or non-printable this time
|| you can type ^@ even or ^S
||this line will give a page break signal for printers
|| cat command is important to check this with -v option
||0 Lets say its last line
Output using cat command with –v option cat -v test.txt
|^@|This is first line^M
|^@| Lets see if ^M is rintable or non-printable this time^M
|^@| you can type ^@ even or ^S^M
|^@|this line will give a page break signal for printers
^M
|^@| cat command is important to check this with -v option^M
|^@|0 Lets say its last line^M
You can observe that ^@ at the beginning and ^M at the last in each line is non-printable character but ^M and ^@ in between was typed by me under vim, so it is printable here. Just observe those two outputs.
Note: It doesn’t show ^L which was there in a file. Because it was a signal for a page break to the printer and it will not hamper the output. That’s we were seeing one blank line in our output whereas in an original file there is no blank line. How we can solve it then?
To Solve ^M at last in each line we can do following things:
Linux Command shell
dos2unix file-name new-file-name
(dos2unix is a Linux utility tool which can convert windows oriented file to Linux compatible, so it will remove ^M automatically from the end of each line
By setting vim configuration file i.e. .vimrc
Open .vimrc which will be at your home location i.e. /home/aliencoders/.vimrc
And save this line set ffs=unix,dos
By editing the file manually . Open test.txt in vim editor and use the following command to substitute in command mode i.e after pressing esc button.
:1,$s/^M//g or :%s/^M//g (Actually ctrl+v and ctrl+ M will show you ^M)
By using strings command in Linux. This command can omit all non-printable characters from the file.
strings file-name > new-file-name
Now, this new-file-name will not contain those non-printable characters.
Tips: To solve other such junks like ^@ either use steps 3 or steps 4
Comments, any other idea to fix it, feedbacks are most welcome!
Need to login on to LINUX machine – but don’t have the privilege? – or want to gain root privilege? Pre Requisite – you need the access physical or command line or console access to reboot the machine.
Follow the steps :-
The very first screen when you restart the linux machine – press an enter or space whichever works and this will take you to the snapshot of step 2.
Fig 1
You will get this screen –
Fig 2
Press Enter. You will get the screen in Fig.3
Fig 3
Press e in the end of 2nd line , the one starting with kernel, to enter the edit mode.
You will get the screen in fig 4, add “s” at the end to login next time in single user mode Type Enter
4. After the step 3 you will get this screen, press b to reboot.
After the reboot, below is the screen you will get-
Now you can change the password by typing passwd,it won’t ask you this time for the old password.
Now, you have successfully changed the root password.
PS : Remove the “s “, that you have put in there previously by using the same procedure. And next time you reboot you can login as root!!
Any query/help/suggestion/feedback regarding this is welcome. Please feel free 🙂
Gnu Privacy Guard or GnuPG or popularly known as GPG is a GPL Licensed alternative to PGP (Pretty Good Privacy) and its openPGP complaint program for *nix people based on rfc 4880. It is part of GNU software project started in 1991 by Werner Koch and majorly funded by German Government. Download its pdf format from here.
Its basic use is to send encrypted mails or files to the recipient who can decrypt these using its private key. It is based on public and private key mechanism for encryption/decryption. We can encrypt any of our data using our own key pair and send it to the person who can read the message if he has the proper key to decrypt it! Many people use public key generated by gpg to verify his email signature too!
It uses following algorithm for various purposes used for safe message communication:
Pubkey: RSA, RSA-E, RSA-S, ELG-E, DSA
Cipher: 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2
Why GPG?
Because it is free and meant to be a replacement of PGP.
Gpg is a CLI program but there are many GUI also through which you can manage keys easily like seahorse for GNOME (yum install seahorse) and KGpg for KDE.
It allows you to encrypt and sign your data, includes a key management system as well as access modules for all kind of public key directories.
If you wish to encrypt your message while sending mail to someone important, you may try this method.
You can share your public key and other users can download it to verify signature in mails/files sent by you for authenticity. It would stop social engineering through email even would stop spams send in the name of your friend’s id.
Applications of GPG
GPG encryption has been added to graphical email client like Evolution for email security.
There is a GNOME front-end application for managing PGP and SSH keys called “Seahorse” which integrates with Nautilus, gedit and Evolution too for encryption, decryption etc.
Enigmail is a data encryption/decryption extension for Mozilla Thunderbird and the SeaMonkey which uses GPG
Mozilla Firefox also gets GPG enabled using Enigform.
GnuPG is being used for Windows Explorer and Outlook through GPG4win tool which are wrapped in the standard Windows installer to make GnuPG easier to get installed and to be used in Windows systems.
It uses hybrid encryption techniques i.e. it uses a combination of symmetric key cryptography for speed and public-key cryptography for easy secured key exchange. By default GnuPG uses the CAST5 symmetrical algorithm.
As a matter of fact, GnuPG does not use patented or otherwise restricted software or algorithms. Instead, GnuPG uses a variety of other, non-patented algorithms.
It will be clearer that how GnuPG works once we see the working of gpg commands step by step:
Which version of gpg we are going to use?
gpg command to generate keys
Analysis of freshly created directory (.gnupg) and files inside it.
Once you get public and private key. You must keep private key safe, once you forget it then you will never be able to decrypt the data. So, better take private key backup.
Want to see the list of public and private keys?
Encrypt the message for specific recipient
Decrypt the encrypted message
GPG commands explained
Which version of gpg we are going to use?
[vim][sjaiswal@AlienCoders ~]$ gpg –version
gpg (GnuPG) 1.4.5
Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.
[vim][sjaiswal@AlienCoders ~/gpg_test]$ gpg –gen-key
gpg (GnuPG) 1.4.5; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.
Please select what kind of key you want:
(1) DSA and Elgamal (default)
(2) DSA (sign only)
(5) RSA (sign only)
Your selection? 1
DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4094
Requested keysize is 4094 bits
rounded up to 4096 bits
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0) 3m
Key expires at Mon 03 Feb 2014 04:46:09 AM MST
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"
Real name: Sanjeev Jaiswal
Email address: sjaiswal@gmail.com
Comment: "GPG Key Test"
You selected this USER-ID:
"Sanjeev Jaiswal ("GPG Key Test") <sjaiswal@gmail.com>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.++++++++++.+++++++++++++++..+++++..++++++++++++++++++++.+++++++++++++++++++++++++.
+gpg: /home/sjaiswal/.gnupg/trustdb.gpg: trustdb created
gpg: key CBE9BE42 marked as ultimately trusted
public and secret key created and signed.
To generate keys using gpg, it would ask which kind of key you wish to use; you can choose any of the given option. Type 1 or 2 or 5.
Type the keysize between the given range
Then provide the expiration date of key. You can use days,weeks, months, years.
Once you are done with expiration days, use the next option carefully. Type Real name, Email and comment appropriately as it will be used while encrypting the data and will ask the recipient name. it will match recipient name before matching the keys.
Then type anything using keyboard, do mouse activities etc to speed up random generation of keys else it may take lot of time.
Once it will get created, .gnupg directory under your home directory will be there. Use ls to see what all files got created.
Analysis of freshly created directory (.gnupg) and files inside it
gpg.conf -> it contains all options set by you. Unless you specify which option file to use (with the command line option "–options filename"), GnuPG uses the file ~/.gnupg/gpg.conf by default. Check strings gpg.conf for more details.
pubring.gpg -> public key stored here. You should export it in ASCII format to send it to others.
pubring.gpg~ -> backup of public key
random_seed -> it contains all random keys used for encryption that you might be typing while generating keys.
secring.gpg -> it’s the secret key ring and one should keep it safe. Better have its backup
trustdb.gpg -> its trusted db which contains signatures, expiration date etc. and from time to time the trust database must be updated so that expired keys or signatures and the resulting changes in the Web of Trust can be tracked.
Normally, GnuPG will calculate when this is required and do it automatically.
Type the message and save it in text file, let’s say message.txt
[vim][sjaiswal@AlienCoders ~/.gnupg]$gpg recipient Sanjeev Jaiswal –encrypt message.txt
[/vim]
It will create message.txt.gpg , which is an encrypted file. To decrypt it, you need to type passphrase that you had typed while generating keys.
Or
[vim]gpg -r real-name –out secrets_to_aliencoders –encrypt secrets
[/vim]
which will have encrypted message in secrets_to_aliencoders
You need a passphrase to unlock the secret key for
user: "Sanjeev Jaiswal ("GPG Key Test") <sjaiswal@gmail.com>"
4096-bit ELG-E key, ID 38765DB9, created 2013-11-05 (main key ID CBE9BE42)
gpg: encrypted with 4096-bit ELG-E key, ID 38765DB9, created 2013-11-05
"Sanjeev Jaiswal ("GPG Key Test") <sjaiswal@gmail.com>"
Hi
This is Sabnjeev
Or
[vim][sjaiswal@AlienCoders ~/.gnupg]$ gpg –output secrets_from_tom –decrypt secrets_to_aliencoders
[/vim]
Which would save the decrypted message in secrets_from_sanjeev
Editing Key
[vim]gpg –edit-key sjaiswal@gmail.com
[/vim]
There is more:
Photo IDs
GnuPG has the ability to add a photo ID to a public key, exactly as in recent Windows versions of PGP. A photo ID attached to a public key can help other users to identify the owner of the key. To add a photo ID to your own public key, use the command "gpg –edit-key <name>" and then enter "addphoto". GnuPG will ask for the filename of a suitable JPEG. No other types of image files can be used.
If you want to see a photo ID on a particular key, enter the command "–show-photos" before using the command "gpg –list-keys <name>". If <name> is omitted, GnuPG will display all the photos (if any) after listing all the keys in your public keyring. Alternatively, if you want photos to be displayed in all cases by default, you should uncomment the line "# show-photos" in the options file inside !GnuPGUser.
Output of trustdb
[vim][Sanjeev@AlienCoders]$ gpg –update-trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 7 trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1 valid: 7 signed: 3 trust: 0-, 0q, 4n, 3m, 0f, 0u
gpg: the next trustdb check will be done on 2014-02-04
[/vim]
The first line shows you the actual trust policy used by your GnuPG installation, and which you can modify at your needs. It states that a key in your keyring is valid if it has been signed by at least 3 marginally trusted keys, or by at least one fully trusted key.
The second line describes the key of level 0, that is the key owned by you. It states that in your keyring you have one level zero key, which is signed by 7 keys. Furthermore among all the level zero keys, you have 0 of them for which you haven't yet evaluated the trust level. 0 of them are the keys for which you have no idea of which validity level to assign (q="I don't know or won't say"). You also have 0 keys that you do not trust at all (n="I do NOT trust"), 0 marginally trusted keys (m="I trust marginally"), 0 fully trusted keys (f="I trust fully") and 1 ultimately trusted keys (u="I trust ultimately").
The third line analyzes the keys of level 1 in your keyring. You have 7 fully valid keys, because you have personally signed them. Furthermore, among the keys that are stored in your keyring, you have 3 of them that are not signed directly by you, but are at least signed by one of the fully valid keys. The trust status counters have the same meaning of the ones in the second line. This time you have 4 keys signed by you but for which you do not trust at all the owner as signer of third party's keys. On the other side, 3 of the 7 keys that you have signed are marginally trusted. This means that you are only marginally confident that the owners of those keys can verify well the keys that they sign.
[vim][Sanjeev@AlienCoders]$ gpg –check-trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 7 trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2014-02-04
[/vim]
If you wish to know other commands that you may use using gpg then try
[vim]man gpg
[/vim] or
[vim]gpg –help
[/vim]
I know it’s not a new topic to discuss and it has lots of online contents already available over the net. But Then I thought it would be useful to this site’s visitors and can have online repository on vim most commonly used commands.
This post has only most commonly used vim commands which we use in our day today development activities. This post will be very helpful for those who wish to learn vim editor from the scratch and it can be useful for all other vim users too.
So, first open the file by using vim filename (single file at one time). I will be posting very soon about manipulation on multiple files using vim editor.
Visitor Rating: 5 Stars
Visitor Rating: 4 Stars