Linux
Re-routing Traffic via a Remote Server
15/02/12 16:18
This isn't a great long guide to using ssh for port forwarding, it just covers the one instance I needed.
Normally I connect my mail program running on c1 (my Mac at home) to server s3 port p3 (my mail server at work).
But today I can't get traffic from c1 to s3 because s3's corporate firewall has blocked me. So I need to re-route all the traffic destined for s3 port p3 via a separate server s2, which fortunately is in another country and I have ssh access to it.
You change your local mail program, instead of connecting to s3 port p3, to connect to "localhost" port p1 (some port on your Mac/PC you haven't used for anything else, e.g. 2000).
Then in a Terminal window you do
ssh -v -L p1:s3:p3@s2
To give you a concrete example, my IMAP server I want to talk to is at imap.work.com port 993.
My remote server (to which I have ssh access) which I want to route all traffic via, is called ssh.mydomain.com and it listens on port 22 as usual. I have an account there with username jkf.
On my local Mac running my Mail client, I've decided to route IMAP traffic via port 2000 as it's an unused port. That "2000" is pretty much any number in the range 1024-65535 that seems to work. If you hit one that doesn't work or produces an error, guess again.
I would type this:
ssh -v -L 2000:imap.work.com:993 jkf@ssh.mydomain.com
If the ssh server on ssh.mydomain.com ran on port 2222 instead of the default port 22, I would just have to add "-p 2222" to that command.
So your traffic from your mail program (set to talk to localhost port p1 which is 2000 in my case) does this:
c0 ==> s1 (==localhost) port p1 ==> s2 port 22 ==> s3 port p3
You will need to set up more than 1 of these as you'll probably want to re-route SMTP traffic as well. So just open another Terminal window and run another similar command in there, for the appropriate SMTP port.
The commands will look like they have just logged you in to your remote ssh server (ssh.mydomain.com) but they will have set up all the magic before they did it. The "-v" in the ssh commands makes it show you more detail of what is going on.
Normally I connect my mail program running on c1 (my Mac at home) to server s3 port p3 (my mail server at work).
But today I can't get traffic from c1 to s3 because s3's corporate firewall has blocked me. So I need to re-route all the traffic destined for s3 port p3 via a separate server s2, which fortunately is in another country and I have ssh access to it.
You change your local mail program, instead of connecting to s3 port p3, to connect to "localhost" port p1 (some port on your Mac/PC you haven't used for anything else, e.g. 2000).
Then in a Terminal window you do
ssh -v -L p1:s3:p3
To give you a concrete example, my IMAP server I want to talk to is at imap.work.com port 993.
My remote server (to which I have ssh access) which I want to route all traffic via, is called ssh.mydomain.com and it listens on port 22 as usual. I have an account there with username jkf.
On my local Mac running my Mail client, I've decided to route IMAP traffic via port 2000 as it's an unused port. That "2000" is pretty much any number in the range 1024-65535 that seems to work. If you hit one that doesn't work or produces an error, guess again.
I would type this:
ssh -v -L 2000:imap.work.com:993 jkf@ssh.mydomain.com
If the ssh server on ssh.mydomain.com ran on port 2222 instead of the default port 22, I would just have to add "-p 2222" to that command.
So your traffic from your mail program (set to talk to localhost port p1 which is 2000 in my case) does this:
c0 ==> s1 (==localhost) port p1 ==> s2 port 22 ==> s3 port p3
You will need to set up more than 1 of these as you'll probably want to re-route SMTP traffic as well. So just open another Terminal window and run another similar command in there, for the appropriate SMTP port.
The commands will look like they have just logged you in to your remote ssh server (ssh.mydomain.com) but they will have set up all the magic before they did it. The "-v" in the ssh commands makes it show you more detail of what is going on.
Comments
Force Linux to Scan for SCSI disks
07/02/12 21:51
Many modern kernels will automatically notice new hard disks being added, and a simple
fdisk -l
will reveal them.
However, if that doesn't list them, do this:
echo "- - -" > /sys/class/scsi_host/host0/scan
echo "- - -" > /sys/class/scsi_host/host1/scan
echo "- - -" > /sys/class/scsi_host/host2/scan
fdisk -l
and you should now see them.
fdisk -l
will reveal them.
However, if that doesn't list them, do this:
echo "- - -" > /sys/class/scsi_host/host0/scan
echo "- - -" > /sys/class/scsi_host/host1/scan
echo "- - -" > /sys/class/scsi_host/host2/scan
fdisk -l
and you should now see them.
Creating Lists in Sympa
22/01/12 18:17
Creating a list in Sympa involves a whole host of files interacting with each other, and all sorts of configuration snippets are used to create the filename of a resource needed to build the list. This attempts to describe what is taken from where, when a list is created from the command-line with a “sympa.pl --create_list” command.
The command (run as root) is
sympa.pl --create_list --input_file=<xml-file>
<xml-file> is a copy of /etc/sympa/sotontemplate.xml with tokens substituted in to set values which are inserted into the template config file. The <xml-file> can be deleted immediately after the list is created, it is not needed.
The <xml-file> looks like this (note the *1* and so on for footnotes):
Notes:
The XML file is meshed into the template config file in /etc/sympa/create_list_templates/<type>/config.tt2 and then interpreted as a TT2 template (see www.tt2.org), and the output goes into the list's own individual config file which is at
/var/lib/sympa/list_data/all.soton.ac.uk/<listname>/config
That file is compiled by Sympa automatically when needed. It can be edited manually at any time, and Sympa will automatically recompile it as and when needed, without you needing to tell it about the change.
Note that the syntax of the template config file config.tt2 is very strict. It is constructed of a series of "paragraphs" which are a bunch of lines, with a blank line terminating the paragraph. It does not support comments, blank lines are *very* important, and leading white space on a line breaks everything. You have been warned!
"Editors" is what Sympa calls list moderators. When they are used depends on the setting of the "send" parameter in the list config file. See the section on "Authorisation Scenarios".
The editors of a list are not statically defined, but are pulled from a SQL database query as needed. The data is cached by Sympa for speed.
If the list has any editors, the editors are defined in the list's config file by a "paragraph" like this:
Notes:
The data source file looks like this:
Note this file *does* handle leading white space on lines.
Most of this is obvious, and it can handle any <db_type> that is available in the Perl DBI modules.
The "source_parameters" from the list's config file are received as the "param" array, and the data source file is again a TT2 file (see www.tt2.org).
The default editor address for every list is “postmaster@ecs.soton.ac.uk".
"Subscribers" are what Sympa calls list members. Only list subscribers receive a copy of postings to the list. So people can be able to post to a list without being able to receive a copy of their own posting. Posters who are subscribers always receive a copy of their own posting.
The subscriber list for a list is generated by an SQL query defined directly in the list's config file, it does not use a .incl file like the editors. The paragraphs defining the subscribers (taken from the template config file, not a list's real config file) look like this:
Notes:
The list of subscribers for each list is cached by Sympa for a length of time defined by the /etc/sympa/sympa.conf setting "default_ttl" and is set to 3600 seconds (1 hour) at the time of writing. So all changes to the table of list subscribers in the "all_lists" database will take up to 1 hour to propagate to the lists. There is no way for force an update of a list's subscribers.
For the purposes of my project, virtually everything is closed and blocked, as users don't need to be able to use the capabilities of the email-driven or web-driven interface to manage lists, as they are all done automatically anyway. The only time authorisation scenari are used is to control who can post to a list.
The people that can post to a list are defined in the list's config file by the paragraph:
/etc/sympa/scenari/send.<scenario-name>
There is also a zero-length file
/etc/sympa/scenari/send.<scenario-name>:ignore
which tells Sympa not to tell anyone about the name or details of the scenario in the web management interface (which we're not using anyway).
The scenario file looks like this:
No comments are allowed. The syntax is strict. Each rule contains a condition, a list of methods, "->" and an action. The "smtp" method is the one we're interested in, it takes the address from the "From:" line of the email message. Rule conditions are matched in order from top to bottom, and the first matching rule is used, at which point rule processing stops.
This returns true or false. If true and the method used is listed, the action happened. You can include single-quoted strings like 'uos-staff', aliases for the name of the list, address of the sender and so on, function calls, all sorts of stuff. The '[sender]' can also be a regexp like '/\@soton\.ac\.uk$/' to cause regexp matches instead of simple string comparison. You can also calls Perl functions, SQL queries, LDAP queries, all sorts of stuff. In a Perl condition, the name of the file is taken from "CustomCondition::<custom-filename>" and the file is located in
/etc/sympa/custom_conditions/<custom-filename>.pm
In this example the list of "Received:" headers is passed to it as a parameter. If it returns 1, the action is executed. If it returns 0, the action is not executed. If it returns undef, there was an error. Changes to the <custom-filename>.pm file only take effect if Sympa is restarted with "service sympa restart".
I have created a send.<scenario-name> file for each type of list we use. Sympa does not notice changes to the scenari unless the datestamp on the list's config file is updated, at which point it updates the scenari for that one list. If you change a scenario file, you have to update the datestamp of the config files for all the lists that use it, forcing on-the-fly recompilation of the lists' config files.
You can change every bit of text ever output by Sympa, and the whole thing is multilingual too. I have stuck with en_US.
All the text output by email to users is contained in the TT2 files in
/etc/sympa/mail_tt2
Changes to files in this directory are automatically noticed by Sympa and processed immediately. Note this does not apply to the file "sympa.po" below.
For example, the "<reason_code>" above, used as a parameter to the "reject()" action, is defined in
/etc/sympa/mail_tt2/authorization_reject.tt2
The best way of finding which file contains the string you are trying to change is to search for a small bit of it in all the mail_tt2/*.tt2 files. Beware the actual text output will not be in these files, but something very close to it will be if you are working in English. Continue reading to find out why...
Let's take the example of the <reason_code> being "send_building". The "reject()" output messages are all defined in the file mentioned above, which is basically a huge "if...then...elsif...elsif...else" statement with hundreds of conditional expressions defined to test the "reason" parameter it is passed by Sympa.
Look through that file and find the "send_building" "ELSIF" condition. That will produce this bit of code:
The actual text output to the user is taken by internationalising the string "send_building", which is passed to the "loc()" function which does the translation. The "loc()" function can have parameters passed to it which are used to insert various bits of text into the translated text. Most of the original text to be translated is *almost* exactly the same as the output of the English translation, but not quite. More fun that way round. A few greps will find the text you're looking to change.
The translation (into better English) is done using the file
/usr/share/locale/en_US/LC_MESSAGES/sympa.mo
This file is a compiled version of the file
/etc/sympa/sympa.po
You can compile it by running
cd /etc/sympa && msgfmt -c -o /usr/share/locale/en_US/LC_MESSAGES/sympa.mo sympa.po
In sympa.po there are multiple definitions which look like this:
service sympa restart
Changes will not take effect until Sympa restarts.
Many mailing lists email all their subscribers automatically at the start of each month telling them they are subscribed to that list, and how to unsubscribe and post successfully. The frequency of such actions are given in the list's config file by the paragraphs:
/etc/sympa/list_task_models/<action>.<task_model>.task
so "expire_task never" is described in
/etc/sympa/list_task_models/expire.never.task
In my project expiry and reminders are not needed nor wanted. So to stop them working, their frequencies have been set to 1000 years. The reason for this is fairly obvious if you look at the structure and syntax of one of the task model files.
The command (run as root) is
sympa.pl --create_list --input_file=<xml-file>
<xml-file> is a copy of /etc/sympa/sotontemplate.xml with tokens substituted in to set values which are inserted into the template config file. The <xml-file> can be deleted immediately after the list is created, it is not needed.
The <xml-file> looks like this (note the *1* and so on for footnotes):
<?xml version="1.0" ?>
<list>
<listname>comp3020</listname> *1*
<type>soton</type> *2*
<listtitle>comp3020</listtitle> *3*
<subject>Students enrolled on COMP3020</subject>
<editor>false</editor>
<replypolicy>other_email</replypolicy>
<replyaddress>god@ecs.soton.ac.uk</replyaddress>
<tag>true</tag>
<ismodule>true</ismodule>
<isbuilding>false</isbuilding>
<isuos>false</isuos>
<owner multiple="1"><email>jkf@ecs.soton.ac.uk</email></owner> *4*
</list>
Notes:
- Sets the name of the list. Must be set in the XML file, not the template config file.
- Sets the value of "type" which is used to determine the location of the template config file used to construct the list's own config file. Must be set in the XML file. Location is /etc/sympa/create_list_templates/<type>/config.tt2
- These all set lower-case-named variables which are referenced in the template config file just mentioned.
- This sets the owner(s) of the list. Must be set in the XML file, not the template config file.
- Any characters such as “&”, "<" and ">" must be encoded as XML entities “&", “<” and “>” or it errors out.
The XML file is meshed into the template config file in /etc/sympa/create_list_templates/<type>/config.tt2 and then interpreted as a TT2 template (see www.tt2.org), and the output goes into the list's own individual config file which is at
/var/lib/sympa/list_data/all.soton.ac.uk/<listname>/config
That file is compiled by Sympa automatically when needed. It can be edited manually at any time, and Sympa will automatically recompile it as and when needed, without you needing to tell it about the change.
Note that the syntax of the template config file config.tt2 is very strict. It is constructed of a series of "paragraphs" which are a bunch of lines, with a blank line terminating the paragraph. It does not support comments, blank lines are *very* important, and leading white space on a line breaks everything. You have been warned!
Editors / Moderators
"Editors" is what Sympa calls list moderators. When they are used depends on the setting of the "send" parameter in the list config file. See the section on "Authorisation Scenarios".
The editors of a list are not statically defined, but are pulled from a SQL database query as needed. The data is cached by Sympa for speed.
If the list has any editors, the editors are defined in the list's config file by a "paragraph" like this:
editor_include
source editors *1*
reception mail
visibility conceal
source_parameters fpas-ecs-sys *2*
Notes:
- The <source> name is used to tell Sympa where to get the SQL query that will produce a list of the email addresses of the editors. The SQL query is defined in a data source file at
/etc/sympa/data_sources/<source>.incl - The comma-separated list of source_parameters are passed into the SQL data source file as param.0, param.1 and so on.
The data source file looks like this:
include_sql_query
db_type mysql
host localhost
db_name all_lists
user secretusername!
passwd secretpassword!
sql_query SELECT email FROM list_moderators WHERE list_address='[% param.0 %]'
Note this file *does* handle leading white space on lines.
Most of this is obvious, and it can handle any <db_type> that is available in the Perl DBI modules.
The "source_parameters" from the list's config file are received as the "param" array, and the data source file is again a TT2 file (see www.tt2.org).
The default editor address for every list is “postmaster@ecs.soton.ac.uk".
Subscribers / Members
"Subscribers" are what Sympa calls list members. Only list subscribers receive a copy of postings to the list. So people can be able to post to a list without being able to receive a copy of their own posting. Posters who are subscribers always receive a copy of their own posting.
The subscriber list for a list is generated by an SQL query defined directly in the list's config file, it does not use a .incl file like the editors. The paragraphs defining the subscribers (taken from the template config file, not a list's real config file) look like this:
user_data_source include2 *1*
include_sql_queryname subscribers *2*
db_type mysql *3*
host localhost
db_name all_lists
user secretusername!
passwd secretpassword!
sql_query SELECT email FROM list_members WHERE list_address='[% listname %]' *4*
Notes:
- This tells Sympa what sort of data source it is. Only "include2" exists in Sympa now.
- Note this is for information only, and has no apparent effect.
- The database type can be anything supported by the Perl DBI modules.
- The [% listname %] is replaced with the name of the list when the list is created, it cannot be written indirectly like that in the list's own config file.
The list of subscribers for each list is cached by Sympa for a length of time defined by the /etc/sympa/sympa.conf setting "default_ttl" and is set to 3600 seconds (1 hour) at the time of writing. So all changes to the table of list subscribers in the "all_lists" database will take up to 1 hour to propagate to the lists. There is no way for force an update of a list's subscribers.
Authorisation Scenari (Who Can Post?)
For the purposes of my project, virtually everything is closed and blocked, as users don't need to be able to use the capabilities of the email-driven or web-driven interface to manage lists, as they are all done automatically anyway. The only time authorisation scenari are used is to control who can post to a list.
The people that can post to a list are defined in the list's config file by the paragraph:
send <scenario-name>
where <scenario-name> tells it to read the rules of the scenario from/etc/sympa/scenari/send.<scenario-name>
There is also a zero-length file
/etc/sympa/scenari/send.<scenario-name>:ignore
which tells Sympa not to tell anyone about the name or details of the scenario in the web management interface (which we're not using anyway).
The scenario file looks like this:
title.gettext Building list, Private, moderated for University staff
CustomCondition::not_from_campus([msg_header->Received]) smtp,dkim,smime,md5 -> reject(reason='from_offcampus')
is_subscriber([listname],[sender]) smtp,dkim,md5,smime -> do_it
is_editor([listname],[sender]) smtp,dkim,md5,smime -> do_it
is_subscriber('uos-staff',[sender]) smtp,dkim,md5,smime -> editorkey
true() smtp,dkim,smime,md5 -> reject(reason='send_building')
No comments are allowed. The syntax is strict. Each rule contains a condition, a list of methods, "->" and an action. The "smtp" method is the one we're interested in, it takes the address from the "From:" line of the email message. Rule conditions are matched in order from top to bottom, and the first matching rule is used, at which point rule processing stops.
Condition
This returns true or false. If true and the method used is listed, the action happened. You can include single-quoted strings like 'uos-staff', aliases for the name of the list, address of the sender and so on, function calls, all sorts of stuff. The '[sender]' can also be a regexp like '/\@soton\.ac\.uk$/' to cause regexp matches instead of simple string comparison. You can also calls Perl functions, SQL queries, LDAP queries, all sorts of stuff. In a Perl condition, the name of the file is taken from "CustomCondition::<custom-filename>" and the file is located in
/etc/sympa/custom_conditions/<custom-filename>.pm
In this example the list of "Received:" headers is passed to it as a parameter. If it returns 1, the action is executed. If it returns 0, the action is not executed. If it returns undef, there was an error. Changes to the <custom-filename>.pm file only take effect if Sympa is restarted with "service sympa restart".
Method
- "smtp" is the one we're interested in. This takes the [sender] from the "From:" line of the incoming email message.
Action
- "do_it" says the operation should be done (in this case, the message should be posted).
- "reject(reason='<reason_code>')" says the message should be rejected and a message sent to the sender explaining why. See the "Responses and Messages" section below for information about the <reason_code>.
- "editorkey" says the message should be saved locally and sent to all the editors of the list with an MD5 checksum. If the editor then sends Sympa the MD5 checksum in an email message telling it to proceed, the message will be posted. If the editor tells Sympa the message should be rejected, it is rejected (don't know what the sender is sent in this case). There are many other actions.
I have created a send.<scenario-name> file for each type of list we use. Sympa does not notice changes to the scenari unless the datestamp on the list's config file is updated, at which point it updates the scenari for that one list. If you change a scenario file, you have to update the datestamp of the config files for all the lists that use it, forcing on-the-fly recompilation of the lists' config files.
Responses and Messages
You can change every bit of text ever output by Sympa, and the whole thing is multilingual too. I have stuck with en_US.
All the text output by email to users is contained in the TT2 files in
/etc/sympa/mail_tt2
Changes to files in this directory are automatically noticed by Sympa and processed immediately. Note this does not apply to the file "sympa.po" below.
For example, the "<reason_code>" above, used as a parameter to the "reject()" action, is defined in
/etc/sympa/mail_tt2/authorization_reject.tt2
The best way of finding which file contains the string you are trying to change is to search for a small bit of it in all the mail_tt2/*.tt2 files. Beware the actual text output will not be in these files, but something very close to it will be if you are working in English. Continue reading to find out why...
Let's take the example of the <reason_code> being "send_building". The "reject()" output messages are all defined in the file mentioned above, which is basically a huge "if...then...elsif...elsif...else" statement with hundreds of conditional expressions defined to test the "reason" parameter it is passed by Sympa.
Look through that file and find the "send_building" "ELSIF" condition. That will produce this bit of code:
[% ELSIF reason == 'send_building' -%]
[%|loc()%]send_building[%END%]
The actual text output to the user is taken by internationalising the string "send_building", which is passed to the "loc()" function which does the translation. The "loc()" function can have parameters passed to it which are used to insert various bits of text into the translated text. Most of the original text to be translated is *almost* exactly the same as the output of the English translation, but not quite. More fun that way round. A few greps will find the text you're looking to change.
The translation (into better English) is done using the file
/usr/share/locale/en_US/LC_MESSAGES/sympa.mo
This file is a compiled version of the file
/etc/sympa/sympa.po
You can compile it by running
cd /etc/sympa && msgfmt -c -o /usr/share/locale/en_US/LC_MESSAGES/sympa.mo sympa.po
In sympa.po there are multiple definitions which look like this:
msgid "send_building"
msgstr "University Building lists are restricted to occupants of the building, and Estates and Facilities staff.\n\n"
"If you think you should be able to post to this list, use http://all.soton.ac.uk/search to search for yourself by staff/student number, and be sure you are sending from the correct address."
- msgid -- This is the text value passed to loc() in the file authorization_reject.tt2 described above.
- msgstr -- This line, and every line starting '"' immediately after it, define the actual text that is sent to the user.
service sympa restart
Changes will not take effect until Sympa restarts.
Regular Tasks
Many mailing lists email all their subscribers automatically at the start of each month telling them they are subscribed to that list, and how to unsubscribe and post successfully. The frequency of such actions are given in the list's config file by the paragraphs:
expire_task never
remind_task never
The frequency in the "<action>_task <task_model>" statement is described in the list task model file/etc/sympa/list_task_models/<action>.<task_model>.task
so "expire_task never" is described in
/etc/sympa/list_task_models/expire.never.task
In my project expiry and reminders are not needed nor wanted. So to stop them working, their frequencies have been set to 1000 years. The reason for this is fairly obvious if you look at the structure and syntax of one of the task model files.
Unable to query the live linux source machine
26/12/10 19:34
You may get this error message from an installation of the VMware Converter Standalone Client or Server.
All you need to do is put a copy of the PuTTY plink.exe and pscp.exe programs into
C:\Program Files (x86)\VMware\VMware vCenter Standalone Converter
All you need to do is put a copy of the PuTTY plink.exe and pscp.exe programs into
C:\Program Files (x86)\VMware\VMware vCenter Standalone Converter
Disable SELinux on RedHat RHEL 6
26/12/10 19:33
Disable IPv6 on RedHat Enterprise RHEL 6
26/12/10 19:32
Add a new file /etc/modprobe.d/ECS.conf containing
alias net-pf-10 off
alias ipv6 off
Edit /etc/sysconfig/network and add a line saying
NETWORKING_IPV6=off
Then do
chkconfig ip6tables off
then reboot.
alias net-pf-10 off
alias ipv6 off
Edit /etc/sysconfig/network and add a line saying
NETWORKING_IPV6=off
Then do
chkconfig ip6tables off
then reboot.
Dropbox
13/05/10 22:21
Update 24 May 2010: I have added some more features to Dropbox and I have released my new “Dropoff” over at www.dropoff.me. Get all the latest and greatest there in future!
I have found a great solution to the common problem of sending and receiving files from other sites and research partners, and generally sending large files around the web where email won’t do the trick.
It’s called “Dropbox” and was originally written by the University of Delaware.
I am launching it as a service at work called “Dropoff” in order that people don’t think you are talking about the service provided by www.dropbox.com which is a totally different thing.
The idea is that you don’t even need to login to send a file to a user within your site/company/University/institution, so external people can use it to send files to people in your institution. It can handle arbitrarily large files, there are no fixed limits. If you log in to the Dropbox website, you can send files to people outside your institution. People who cannot log in can only send files to people within your institution. This stops the rest of the world using it to send people to other people who aren’t members of your institution.
I have added various extra features to it:
I have customised it quite a bit just for our site, so if you want a copy of my patched version, along with a guide as to what changes I have made, then please contact me.
I have found a great solution to the common problem of sending and receiving files from other sites and research partners, and generally sending large files around the web where email won’t do the trick.
It’s called “Dropbox” and was originally written by the University of Delaware.
I am launching it as a service at work called “Dropoff” in order that people don’t think you are talking about the service provided by www.dropbox.com which is a totally different thing.
The idea is that you don’t even need to login to send a file to a user within your site/company/University/institution, so external people can use it to send files to people in your institution. It can handle arbitrarily large files, there are no fixed limits. If you log in to the Dropbox website, you can send files to people outside your institution. People who cannot log in can only send files to people within your institution. This stops the rest of the world using it to send people to other people who aren’t members of your institution.
I have added various extra features to it:
- Active Directory AD authentication (to multiple AD sites at once if needed)
- Virus scanning of uploaded files, using ClamAV
I have customised it quite a bit just for our site, so if you want a copy of my patched version, along with a guide as to what changes I have made, then please contact me.
Resending Unix Mbox Files
25/03/10 15:12
Unfortunately someone screwed up the installation of one of our servers so that mail to local addresses was being delivered into /var/spool/mail/ instead of being sent onwards to our SMTP server.
Getting the sendmail.mc correct was the easy bit, there is a simple “null client” sendmail.mc file I wrote years ago which does that nicely.
The hard bit was taking all the Unix mbox files in /var/spool/mail and /var/mail and re-delivering them all to their intended recipients. The bit most people get wrong is the separator between messages. The separator is not a line starting with “From “. The separator is a blank line followed by a line starting with “From “. So I wrote my own script to do it which you are very welcome to download and use.
Getting the sendmail.mc correct was the easy bit, there is a simple
The hard bit was taking all the Unix mbox files in /var/spool/mail and /var/mail and re-delivering them all to their intended recipients. The bit most people get wrong is the separator between messages. The separator is not a line starting with “From “. The separator is a blank line followed by a line starting with “From “. So I wrote my own script to do it which you are very welcome to download and use.
VMware Converter - Converting an Existing Windows System
18/10/09 20:23
The Converter is a client/server program, where the converter server runs on ecsvm-admin.ecs and the client can run anywhere, such as win-admin.
Firewall Rules
If the source physical Windows system is in the DMZ, you will need to add a couple of rules to the firewall "Short-Term Rules" section allowing all traffic from ECS-internal to the source host and from the source host to ECS-internal. You should remove these rules again once the conversion has completed.
Starting the Conversion
Start the VMWare Converter Standalone Client program.
IP Address/Name = ecsvm-admin:7443
Username = administrator (or your own sys* username if you are registered on the vSphere with an account).
Password = Windows-infrastructure password
Click the "Convert Machine" button in the toolbar.
Source type = Powered-on machine
Give remote machine full hostname.
Give username and password of a user in the "Administrators" group on the source Windows system, such as the "ECS2000\administrator" account and domain password..
OS Family = Windows.
Next.
It should connect to the source physical system.
If it fails, then give it the IP address of the source physical system instead of the name of it.
If it still fails, then Remote Desktop to the source physical system, download the converter (from browsing the datastores starting at http://ecsvm-admin.ecs.soton.ac.uk/) and install the converter agent (not the client/server setup) onto the source system, then run it again from there.
When it asks for the name of the source system to convert, give it the IP address of the source system (which is the same system you are now running the Converter on).
Tell it to automatically uninstall the files when the import succeeds.
It will then deploy the agent, which takes a few seconds.
VMware Infrastructure server details:
Server = ecsvm-admin
Username = administrator (or your sys* username if you have an account on vSphere).
Password = Windows-infrastructure password (or yours).
Next.
Select host to run the VM on = ecsvm-admin1.ecs.soton.ac.uk.
Virtual machine name = short hostname (eg. major-backup).
Datastore = infrastore1-Vol2 (an infrastore volume with plenty of free space).
Virtual machine version = Version 7.
Next.
Options
Destination Attributes: Name = short hostname, Folder = ecs
Data to copy: Click "Advanced...", then "Target Layout...".
Set all large, non-swap filesystems to "Thin provisioning" (this is "Dynamic disks" in Microsoft speak, where only the disk space in use is actually allocated on disk, used disk space expands as necessary to hold the data, up to the maximum set by the size of the filesystem).
Set the Size of the normal large filesystems (eg. "C:") to a reasonable number, no point in making them huge, most will fit in 60Gbytes.
Switch to the "Source Volumes" tab to see how much space is actually in use at the moment.
Normally set C: to thin provisioning, 60GBytes or more.
It is important not to waste disk space on Flat-provisioned disks that are not going to use all their space. Disk is relatively expensive.
Devices: Numer of processors = 1.
Disk controller = SCSI LSI Logic or SCSI Buslogic.
Memory = usually the default will be fine.
Networks: Network adapters to connect = 1
Set the network for the network adapter to be the same Virtual Machine Network VLAN as the physical machine you are converting.
The new virtual machine will take over the IP address of the physical source machine.
Services: Source Services: switch off services that should not be left running when the VM is created (such as SQL Server or WWW Publishing Service, and Hyper-V services if moving from Hyper-V to VMware). Destination Services: set all Hyper-V services to "Disabled".
Advanced Options: Power on target machine = yes
Power off source machine = yes
Install VMware Tools on the imported virtual machine = yes
Configure guest preferences for the virtual machine = no
Remove System Restore checkpoints on destination = yes
Reconfigure destination virtual machine = yes
Next.
It should now just show you the final option settings and then start the process of converting the host. Wait for the whole process to finish before touching either the source (physical) or destination (virtual) machines.
How long it takes depends on the quantity of data that has to be moved. You can expect about 20 to 25 MBytes/second conversion speed. A machine with about 7GB of used disk takes about 25 minutes to convert.
As the process is run by the Converter Server (running on ecsvm-admin), it doesn't matter if you quit the Windows app used to watch the process, you can track the progress of the conversion from the Converter Windows app running on any PC.
After the Conversion has Finished
Open a console on the new VM.
In the "VM/Guest" menu, install the VM tools. This will insert a CD into the virtual CD drive, what happens then is dependent on the Windows Autoplay preferences in the virtual machine.
This will force a restart of the virtual machine.
Shutdown the virtual machine.
Once the VM has stopped, edit the settings of the VM and choose the middle "Options" tab. In the "VMware Tools" settings, right at the bottom right of the dialog there is an option to "Synchronize guest time with host". Tick this box.
Okay that, then power on the VM.
It should successfully boot.
The VM should now be fully running happily and serving its services to the users.
Now just remove the two temporary firewall rules you added at the start of this process.
Firewall Rules
If the source physical Windows system is in the DMZ, you will need to add a couple of rules to the firewall "Short-Term Rules" section allowing all traffic from ECS-internal to the source host and from the source host to ECS-internal. You should remove these rules again once the conversion has completed.
Starting the Conversion
Start the VMWare Converter Standalone Client program.
IP Address/Name = ecsvm-admin:7443
Username = administrator (or your own sys* username if you are registered on the vSphere with an account).
Password = Windows-infrastructure password
Click the "Convert Machine" button in the toolbar.
Source type = Powered-on machine
Give remote machine full hostname.
Give username and password of a user in the "Administrators" group on the source Windows system, such as the "ECS2000\administrator" account and domain password..
OS Family = Windows.
Next.
It should connect to the source physical system.
If it fails, then give it the IP address of the source physical system instead of the name of it.
If it still fails, then Remote Desktop to the source physical system, download the converter (from browsing the datastores starting at http://ecsvm-admin.ecs.soton.ac.uk/) and install the converter agent (not the client/server setup) onto the source system, then run it again from there.
When it asks for the name of the source system to convert, give it the IP address of the source system (which is the same system you are now running the Converter on).
Tell it to automatically uninstall the files when the import succeeds.
It will then deploy the agent, which takes a few seconds.
VMware Infrastructure server details:
Server = ecsvm-admin
Username = administrator (or your sys* username if you have an account on vSphere).
Password = Windows-infrastructure password (or yours).
Next.
Select host to run the VM on = ecsvm-admin1.ecs.soton.ac.uk.
Virtual machine name = short hostname (eg. major-backup).
Datastore = infrastore1-Vol2 (an infrastore volume with plenty of free space).
Virtual machine version = Version 7.
Next.
Options
Destination Attributes: Name = short hostname, Folder = ecs
Data to copy: Click "Advanced...", then "Target Layout...".
Set all large, non-swap filesystems to "Thin provisioning" (this is "Dynamic disks" in Microsoft speak, where only the disk space in use is actually allocated on disk, used disk space expands as necessary to hold the data, up to the maximum set by the size of the filesystem).
Set the Size of the normal large filesystems (eg. "C:") to a reasonable number, no point in making them huge, most will fit in 60Gbytes.
Switch to the "Source Volumes" tab to see how much space is actually in use at the moment.
Normally set C: to thin provisioning, 60GBytes or more.
It is important not to waste disk space on Flat-provisioned disks that are not going to use all their space. Disk is relatively expensive.
Devices: Numer of processors = 1.
Disk controller = SCSI LSI Logic or SCSI Buslogic.
Memory = usually the default will be fine.
Networks: Network adapters to connect = 1
Set the network for the network adapter to be the same Virtual Machine Network VLAN as the physical machine you are converting.
The new virtual machine will take over the IP address of the physical source machine.
Services: Source Services: switch off services that should not be left running when the VM is created (such as SQL Server or WWW Publishing Service, and Hyper-V services if moving from Hyper-V to VMware). Destination Services: set all Hyper-V services to "Disabled".
Advanced Options: Power on target machine = yes
Power off source machine = yes
Install VMware Tools on the imported virtual machine = yes
Configure guest preferences for the virtual machine = no
Remove System Restore checkpoints on destination = yes
Reconfigure destination virtual machine = yes
Next.
It should now just show you the final option settings and then start the process of converting the host. Wait for the whole process to finish before touching either the source (physical) or destination (virtual) machines.
How long it takes depends on the quantity of data that has to be moved. You can expect about 20 to 25 MBytes/second conversion speed. A machine with about 7GB of used disk takes about 25 minutes to convert.
As the process is run by the Converter Server (running on ecsvm-admin), it doesn't matter if you quit the Windows app used to watch the process, you can track the progress of the conversion from the Converter Windows app running on any PC.
After the Conversion has Finished
Open a console on the new VM.
In the "VM/Guest" menu, install the VM tools. This will insert a CD into the virtual CD drive, what happens then is dependent on the Windows Autoplay preferences in the virtual machine.
This will force a restart of the virtual machine.
Shutdown the virtual machine.
Once the VM has stopped, edit the settings of the VM and choose the middle "Options" tab. In the "VMware Tools" settings, right at the bottom right of the dialog there is an option to "Synchronize guest time with host". Tick this box.
Okay that, then power on the VM.
It should successfully boot.
The VM should now be fully running happily and serving its services to the users.
Now just remove the two temporary firewall rules you added at the start of this process.
VMware Converter - Converting an Existing Linux System
18/10/09 20:22
The Converter is a client/server program, where the converter server runs on ecsvm-admin.ecs and the client can run anywhere, such as win-admin.
Firewall Rules
If the source physical Linux system is in the DMZ, you will need to add a couple of rules to the firewall "Short-Term Rules" section allowing all traffic from ECS-internal to the source host and from the source host to ECS-internal. You should remove these rules again once the conversion has completed.
Before You Start
On the final virtual machine, you will need to install the VMware tools and set the VM settings to synchronise the time on the VM with that of the ecsvm-admin server. So you won't want ntpd to be running. Also, you are going to need to reboot the VM at least once after you have converted it, so stop and disable the primary user services (e.g. httpd, mysqld) that are running on the server. You can start up the user services again as the last step after getting the VM Linux system running. So I would start with
service ntpd stop
service httpd stop
service mysqld stop
chkconfig ntpd off
chkconfig httpd off
chkconfig mysqld off
Also, if the physical source machine is actually a Windows 2008 Hyper-V VM, then you will want to do the same to the "inputvsc" service, and copy the seth0 device settings to eth0 (remembering to change the device name in /etc/sysconfig/network-scripts/ifcfg-eth0 after copying the ifcfg-seth0 file onto it) as VMware will use the eth0 device and not the seth0 device.
Once running in VMware, the kernel will need to be able to "probe" the disk controllers in order to be able to find the controller types. So in /boot/grub/grub.conf, edit the kernel command line arguments and remove any settings that set "hda=noprobe" or similar. You don't need to reboot after this change, this will be applied once the VM conversion has finished and VMware starts the new VM.
Starting the Conversion
Start the VMWare Converter Standalone Client program.
IP Address/Name = ecsvm-admin:7443
Username = administrator (or your own sys* username if you are registered on the vSphere with an account).
Password = Windows-infrastructure password
Click the "Convert Machine" button in the toolbar.
Source type = Powered-on machine
Give remote machine full hostname.
Give root username and password.
Next.
VMware Infrastructure server details:
Server = ecsvm-admin
Username = administrator (or your sys* username if you have an account on vSphere).
Password = Windows-infrastructure password (or yours).
Next.
Select host to run the VM on = ecsvm-admin1.ecs.soton.ac.uk.
Virtual machine name = short hostname (eg. gander).
Datastore = infrastore1-Vol2 (an infrastore volume with plenty of free space).
Virtual machine version = Version 7.
Next.
Options
Destination Attributes: Name = short hostname, Folder = ecs
Data to copy: Click "Advanced...", then "Target Layout...".
Set all large, non-swap filesystems to "Thin provisioning" (this is "Dynamic disks" in Microsoft speak, where only the disk space in use is actually allocated on disk, used disk space expands as necessary to hold the data, up to the maximum set by the size of the filesystem).
Set the Size of the normal large filesystems (eg. "/") to a reasonable number, no point in making them huge, most will fit in 40Gbytes.
Switch to the "Source Volumes" tab to see how much space is actually in use at the moment.
Normally set / to thin provisioning, 40GBytes, leave /boot and swap alone.
Devices: Numer of processors = 1.
Disk controller = SCSI LSI Logic or SCSI Buslogic.
Memory = usually the default will be fine.
Networks: Network adapters to connect = 1
Set the network for the network adapter to be the same Virtual Machine Network VLAN as the physical machine you are converting.
The new virtual machine will take over the IP address of the physical source machine.
Advanced Options: Power on target machine = yes
Power off source machine = yes
Reconfigure destination virtual machine = yes
Helper VM Network: Look up the IP address of the host "ecsvm--helper" which is a spare IP address unused by anything else in the same "" as the physical host you are converting. There are currently hosts defined such as "ecsvm-systems-helper", "ecsvm-dmz-helper" and "ecsvm-servers-helper". The IP configuration parameters and DNS setup for the Helper VM Network must all be consistent with its IP address. This "helper" machine is a temporary system setup by the VMware Converter just for use during the process of converting a Linux box, it disappears again at the end.
Next.
It should now just show you the final option settings and then start the process of converting the host. Wait for the whole process to finish before touching either the source (physical) or destination (virtual) machines.
How long it takes depends on the quantity of data that has to be moved. You can expect about 20 to 25 MBytes/second conversion speed. A machine with about 7GB of used disk takes about 25 minutes to convert.
As the process is run by the Converter Server (running on ecsvm-admin), it doesn't matter if you quit the Windows app used to watch the process, you can track the progress of the conversion from the Converter Windows app running on any PC.
After the Conversion has Finished
Open a console on the new VM.
In the "VM/Guest" menu, install the VM tools.
mount /dev/cdrom /mnt
cd /tmp
tar xzf /mnt/VM*
umount /mnt
cd vmware*
./vmware-tools-install.pl
Accept all the defaults. It will find suitable modules for your kernel, or else will compile them itself, and build a new initrd, grub.conf and modprobe.conf for your kernel.
Assuming that succeeded, then
shutdown -h now
Once the VM has stopped, edit the settings of the VM and choose the middle "Options" tab. In the "VMware Tools" settings, right at the bottom right of the dialog there is an option to "Synchronize guest time with host". Tick this box.
Okay that, then power on the VM.
It should successfully boot.
The only remaining tasks are to enable and start up the user services, but not ntpd.
chkconfig mysqld on
chkconfig httpd on
service mysqld start
service httpd start
The VM should now be fully running happily and serving its services to the users. If you prefer, you can always reboot the VM again instead of running those last two "service ... start" commands to prove it is all okay and booting correctly.
Now just remove the two temporary firewall rules you added at the start of this process.
Firewall Rules
If the source physical Linux system is in the DMZ, you will need to add a couple of rules to the firewall "Short-Term Rules" section allowing all traffic from ECS-internal to the source host and from the source host to ECS-internal. You should remove these rules again once the conversion has completed.
Before You Start
On the final virtual machine, you will need to install the VMware tools and set the VM settings to synchronise the time on the VM with that of the ecsvm-admin server. So you won't want ntpd to be running. Also, you are going to need to reboot the VM at least once after you have converted it, so stop and disable the primary user services (e.g. httpd, mysqld) that are running on the server. You can start up the user services again as the last step after getting the VM Linux system running. So I would start with
service ntpd stop
service httpd stop
service mysqld stop
chkconfig ntpd off
chkconfig httpd off
chkconfig mysqld off
Also, if the physical source machine is actually a Windows 2008 Hyper-V VM, then you will want to do the same to the "inputvsc" service, and copy the seth0 device settings to eth0 (remembering to change the device name in /etc/sysconfig/network-scripts/ifcfg-eth0 after copying the ifcfg-seth0 file onto it) as VMware will use the eth0 device and not the seth0 device.
Once running in VMware, the kernel will need to be able to "probe" the disk controllers in order to be able to find the controller types. So in /boot/grub/grub.conf, edit the kernel command line arguments and remove any settings that set "hda=noprobe" or similar. You don't need to reboot after this change, this will be applied once the VM conversion has finished and VMware starts the new VM.
Starting the Conversion
Start the VMWare Converter Standalone Client program.
IP Address/Name = ecsvm-admin:7443
Username = administrator (or your own sys* username if you are registered on the vSphere with an account).
Password = Windows-infrastructure password
Click the "Convert Machine" button in the toolbar.
Source type = Powered-on machine
Give remote machine full hostname.
Give root username and password.
Next.
VMware Infrastructure server details:
Server = ecsvm-admin
Username = administrator (or your sys* username if you have an account on vSphere).
Password = Windows-infrastructure password (or yours).
Next.
Select host to run the VM on = ecsvm-admin1.ecs.soton.ac.uk.
Virtual machine name = short hostname (eg. gander).
Datastore = infrastore1-Vol2 (an infrastore volume with plenty of free space).
Virtual machine version = Version 7.
Next.
Options
Destination Attributes: Name = short hostname, Folder = ecs
Data to copy: Click "Advanced...", then "Target Layout...".
Set all large, non-swap filesystems to "Thin provisioning" (this is "Dynamic disks" in Microsoft speak, where only the disk space in use is actually allocated on disk, used disk space expands as necessary to hold the data, up to the maximum set by the size of the filesystem).
Set the Size of the normal large filesystems (eg. "/") to a reasonable number, no point in making them huge, most will fit in 40Gbytes.
Switch to the "Source Volumes" tab to see how much space is actually in use at the moment.
Normally set / to thin provisioning, 40GBytes, leave /boot and swap alone.
Devices: Numer of processors = 1.
Disk controller = SCSI LSI Logic or SCSI Buslogic.
Memory = usually the default will be fine.
Networks: Network adapters to connect = 1
Set the network for the network adapter to be the same Virtual Machine Network VLAN as the physical machine you are converting.
The new virtual machine will take over the IP address of the physical source machine.
Advanced Options: Power on target machine = yes
Power off source machine = yes
Reconfigure destination virtual machine = yes
Helper VM Network: Look up the IP address of the host "ecsvm-
Next.
It should now just show you the final option settings and then start the process of converting the host. Wait for the whole process to finish before touching either the source (physical) or destination (virtual) machines.
How long it takes depends on the quantity of data that has to be moved. You can expect about 20 to 25 MBytes/second conversion speed. A machine with about 7GB of used disk takes about 25 minutes to convert.
As the process is run by the Converter Server (running on ecsvm-admin), it doesn't matter if you quit the Windows app used to watch the process, you can track the progress of the conversion from the Converter Windows app running on any PC.
Open a console on the new VM.
In the "VM/Guest" menu, install the VM tools.
mount /dev/cdrom /mnt
cd /tmp
tar xzf /mnt/VM*
umount /mnt
cd vmware*
./vmware-tools-install.pl
Accept all the defaults. It will find suitable modules for your kernel, or else will compile them itself, and build a new initrd, grub.conf and modprobe.conf for your kernel.
Assuming that succeeded, then
shutdown -h now
Once the VM has stopped, edit the settings of the VM and choose the middle "Options" tab. In the "VMware Tools" settings, right at the bottom right of the dialog there is an option to "Synchronize guest time with host". Tick this box.
Okay that, then power on the VM.
It should successfully boot.
The only remaining tasks are to enable and start up the user services, but not ntpd.
chkconfig mysqld on
chkconfig httpd on
service mysqld start
service httpd start
The VM should now be fully running happily and serving its services to the users. If you prefer, you can always reboot the VM again instead of running those last two "service ... start" commands to prove it is all okay and booting correctly.
Now just remove the two temporary firewall rules you added at the start of this process.
Install VMware Tools on a Linux Client
23/09/09 20:13
First, log in to the VM as root and "yum update".
Then "yum install gcc make binutils kernel-devel kernel-headers".
Then reboot the VM.
Then start up the vSphere Client, right click on the VM and choose Guest - Install VMware Tools.
Back into the VM's root session.
mount /dev/cdrom /mnt
cd /tmp
tar xzf /mnt/VM*
umount /mnt
cd vmware-tools-distrib
./vmware-install.pl
Accept all the defaults, and let it do everything it wants. If you have a fully updated and correct system, it should install flawlessly.
Check the "ifconfig -a" and ensure that all the network devices that exist have startup scripts in /etc/sysconfig/network-scripts/ifcfg-eth*.
Go into the vSphere Client again and right-click on the VM. Look in the "Guest" sub-menu and tell it to stop installing the VMware Tools if it offers you that.
Reboot the VM.
If you have problems...
The most likely problem is that you are running an el5xen kernel or some other xen kernel, which you don't want to be doing.
Once you've done a yum update, take the xen kernel you are running (uname -a will tell you) and do something like this:
yum install kernel-2.6.18-164.el5
Then edit /boot/grub/grub.conf and ensure that the "default=" setting at the top is set to boot the kernel you just installed and not the xen kernel (they start numbering from the top of the file from 0).
Then reboot so you are running the non-xen kernel.
Then
yum install kernel-devel-2.6.18-164.el5
Then re-run /tmp/vmware-tools-distrib/vmware-install.pl and if it asks you for the kernel headers location, give it
/usr/src/kernels/2.6.18-164.el5/include
But you will most likely find that it just happily works on its own!
Then just reboot to pick up all the VMware tools in a fresh boot.
Then "yum install gcc make binutils kernel-devel kernel-headers".
Then reboot the VM.
Then start up the vSphere Client, right click on the VM and choose Guest - Install VMware Tools.
Back into the VM's root session.
mount /dev/cdrom /mnt
cd /tmp
tar xzf /mnt/VM*
umount /mnt
cd vmware-tools-distrib
./vmware-install.pl
Accept all the defaults, and let it do everything it wants. If you have a fully updated and correct system, it should install flawlessly.
Check the "ifconfig -a" and ensure that all the network devices that exist have startup scripts in /etc/sysconfig/network-scripts/ifcfg-eth*.
Go into the vSphere Client again and right-click on the VM. Look in the "Guest" sub-menu and tell it to stop installing the VMware Tools if it offers you that.
Reboot the VM.
If you have problems...
The most likely problem is that you are running an el5xen kernel or some other xen kernel, which you don't want to be doing.
Once you've done a yum update, take the xen kernel you are running (uname -a will tell you) and do something like this:
yum install kernel-2.6.18-164.el5
Then edit /boot/grub/grub.conf and ensure that the "default=" setting at the top is set to boot the kernel you just installed and not the xen kernel (they start numbering from the top of the file from 0).
Then reboot so you are running the non-xen kernel.
Then
yum install kernel-devel-2.6.18-164.el5
Then re-run /tmp/vmware-tools-distrib/vmware-install.pl and if it asks you for the kernel headers location, give it
/usr/src/kernels/2.6.18-164.el5/include
But you will most likely find that it just happily works on its own!
Then just reboot to pick up all the VMware tools in a fresh boot.
VMware Converter Fails for Linux Client
23/09/09 20:11
You need to change the installed SCSI controller to be a LSI Logic controller. The original machine may have been configured for a Transtec 3Ware SCSI controller.
The main relevant article is here:
http://tipstricksandmore.blogspot.com/2009/01/after-converting-physical-rhel4-system.html
Once the VM has been converted, you can change the hardware of the VM in the "Edit Settings..." menu from right-clicking on the VM. You can only change it when the VM is powered off.
Edit the VM settings and connect the DVD drive to a "Datastore ISO File": infra1-localDisk/vSphere Management Assistant/rhel-5-server-i386-dvd.iso.
Set the Device Status to "Connected" and "Connect at power on".
In the VM options page, set it to go into the BIOS setup at next boot.
Open a console on the VM and power it on. It will go into the BIOS setup.
In the "Boot" BIOS menu, select the DVD drive and press + to move it to the top.
Save and exit the BIOS setup.
It will boot from DVD.
Enter "linux rescue".
You don't need any network interfaces.
Let it look for the installed system to mount under /mnt/sysimage.
chroot /mnt/sysimage
Replace hda with sda in /etc/fstab, /boot/grub/device.map and /boot/grub/grub.conf
grub-install /dev/sda
Make sure /etc/modules.conf is empty or non-existent.
Edit /etc/modprobe.conf and set
alias eth0 pcnet32
alias eth1 pcnet32
alias scsi_hostadapter mptbase
alias scsi_hostadapter1 mptscsih
Work out the full path to the initrd image you are going to rebuild. They are in /boot and are called initrd*.
The /boot/grub/grub.conf will point to the right one.
So in my example it is "/boot/initrd-2.6.18-128.7.1.el5xen.img".
There should be a directory under /lib/modules called the same version number.
Then you use a command like this to rebuild it
mkinitrd -v -f /boot/initrd-2.6.18-128.7.1.el5xen.img 2.6.18-128.7.1.el5xen
exit
reboot
Press Esc to get the boot menu and force it to boot from the hard disk.
Hopefully it will boot this time!
Shut it down again, edit the VM settings and set the DVD drive back to "Client Device".
Boot your VM normally.
The main relevant article is here:
http://tipstricksandmore.blogspot.com/2009/01/after-converting-physical-rhel4-system.html
Once the VM has been converted, you can change the hardware of the VM in the "Edit Settings..." menu from right-clicking on the VM. You can only change it when the VM is powered off.
Edit the VM settings and connect the DVD drive to a "Datastore ISO File": infra1-localDisk/vSphere Management Assistant/rhel-5-server-i386-dvd.iso.
Set the Device Status to "Connected" and "Connect at power on".
In the VM options page, set it to go into the BIOS setup at next boot.
Open a console on the VM and power it on. It will go into the BIOS setup.
In the "Boot" BIOS menu, select the DVD drive and press + to move it to the top.
Save and exit the BIOS setup.
It will boot from DVD.
Enter "linux rescue".
You don't need any network interfaces.
Let it look for the installed system to mount under /mnt/sysimage.
chroot /mnt/sysimage
Replace hda with sda in /etc/fstab, /boot/grub/device.map and /boot/grub/grub.conf
grub-install /dev/sda
Make sure /etc/modules.conf is empty or non-existent.
Edit /etc/modprobe.conf and set
alias eth0 pcnet32
alias eth1 pcnet32
alias scsi_hostadapter mptbase
alias scsi_hostadapter1 mptscsih
Work out the full path to the initrd image you are going to rebuild. They are in /boot and are called initrd*.
The /boot/grub/grub.conf will point to the right one.
So in my example it is "/boot/initrd-2.6.18-128.7.1.el5xen.img".
There should be a directory under /lib/modules called the same version number.
Then you use a command like this to rebuild it
mkinitrd -v -f /boot/initrd-2.6.18-128.7.1.el5xen.img 2.6.18-128.7.1.el5xen
exit
reboot
Press Esc to get the boot menu and force it to boot from the hard disk.
Hopefully it will boot this time!
Shut it down again, edit the VM settings and set the DVD drive back to "Client Device".
Boot your VM normally.
Mouse support in Hyper-V
18/04/09 20:30
This is how to add mouse support to the LICs (Linux Integration Components) provided by Microsoft. This does not involve using the Beta version of the LICs at all, it is all done with the production release version.
Firstly install everything else involved in getting the LICs working. There is another article in this blog that will explain how to get the LICs working with RedHat or CentOS 5.2.
On your Windows 2008 or Hyper-V server, download the “inputvsc.iso” from http://www.xen.org/download/satori.html.
Using the Hyper-V “Connect” window and its Media menu, set the DVD drive to point to the “inputvsc.iso” you just downloaded.
In the virtual machine, “mkdir /mnt” and “mount /dev/hdc /mnt”. That should mount the ISO on /mnt.
Make somewhere to put it, such as “mkdir -p /opt/inputdriver”.
Copy the contents of the ISO to there, “cp -pr /mnt/* /opt/inputdriver”.
Unmount the ISO, “umount /mnt”.
Eject the media using the Media menu in the Hyper-V “Connect” window.
Go into that directory, “cd /opt/inputdriver”.
Install one required package, “yum install xorg-x11-server-sdk”.
Install the mouse driver, “perl setup.pl inputdriver”.
That should succeed.
Start X Windows with “startx” to test it. If the mouse support doesn’t work, you can always kill X by pressing Ctrl-Alt-Backspace, or by logging in remotely to the virtual machine as root and typing “killall X”.
That’s about it. It worked fine for me!
Firstly install everything else involved in getting the LICs working. There is another article in this blog that will explain how to get the LICs working with RedHat or CentOS 5.2.
On your Windows 2008 or Hyper-V server, download the “inputvsc.iso” from http://www.xen.org/download/satori.html.
Using the Hyper-V “Connect” window and its Media menu, set the DVD drive to point to the “inputvsc.iso” you just downloaded.
In the virtual machine, “mkdir /mnt” and “mount /dev/hdc /mnt”. That should mount the ISO on /mnt.
Make somewhere to put it, such as “mkdir -p /opt/inputdriver”.
Copy the contents of the ISO to there, “cp -pr /mnt/* /opt/inputdriver”.
Unmount the ISO, “umount /mnt”.
Eject the media using the Media menu in the Hyper-V “Connect” window.
Go into that directory, “cd /opt/inputdriver”.
Install one required package, “yum install xorg-x11-server-sdk”.
Install the mouse driver, “perl setup.pl inputdriver”.
That should succeed.
Start X Windows with “startx” to test it. If the mouse support doesn’t work, you can always kill X by pressing Ctrl-Alt-Backspace, or by logging in remotely to the virtual machine as root and typing “killall X”.
That’s about it. It worked fine for me!
Hyper-V CentOS 5.2 Distributions
28/12/08 09:59
Update: 22nd May 2009 - This does not work with CentOS 5.3 or RedHat 5.3.
Update: 29th December 2008 - I have compacted the original vhd files rather better, and the total download for each version is now about 2.3 Gbytes.
To make life easier for everyone, I have put together a couple of VHD files for Hyper-V that contain a pre-built x86 and x86_64 (x64) distribution of CentOS 5.2 including pre-installed Linux Integration Components. They are fully patched up to date, including the latest kernel version available, as of Christmas 2008.
The root password for each of them is the word “password” (without the quotes).
To construct each one, go to the relevant directories for
the x86 (32-bit)
or x64 (64-bit)
versions and download all the zip files in the directory.
Unpack each zip file and you will have a string of files partaa, partab, partac and so on.
On a Windows system you can join these together into the .vhd file with the command (in a normal Command Prompt window)
x86: copy /B partaa+partab+partac+partad+partae+partaf+partag+partah+partai+partaj+partak+partal CentOS5.2.x86.vhd
(all of that should be on one line)
x64: copy /B partaa+partab+partac+partad+partae+partaf+partag+partah+partai+partaj+partak+partal CentOS5.2.x64.vhd
(all of that should be on one line)
You should end up with a single .vhd file with the following size:
x86: 16173279232 bytes
x64: 18433592832 bytes
Then just build a virtual machine around each one with the .vhd file as the IDE hard disk, and with a Network Adapter (not a Legacy Network Adapter) in it.
Remember that the root password is the word “password” (without the quotes).
You will need to edit these files
and then run the command “setup” to configure the IP address and so on of the “seth0” network device. Then just reboot and you’re away.
If you need a graphical interface with a mouse and such, then change the “3” to a “5” in the “default” line in /etc/inittab and reboot. But I would advise leaving it in text-only mode.
Update: 29th December 2008 - I have compacted the original vhd files rather better, and the total download for each version is now about 2.3 Gbytes.
To make life easier for everyone, I have put together a couple of VHD files for Hyper-V that contain a pre-built x86 and x86_64 (x64) distribution of CentOS 5.2 including pre-installed Linux Integration Components. They are fully patched up to date, including the latest kernel version available, as of Christmas 2008.
The root password for each of them is the word “password” (without the quotes).
To construct each one, go to the relevant directories for
the x86 (32-bit)
or x64 (64-bit)
versions and download all the zip files in the directory.
Unpack each zip file and you will have a string of files partaa, partab, partac and so on.
On a Windows system you can join these together into the .vhd file with the command (in a normal Command Prompt window)
x86: copy /B partaa+partab+partac+partad+partae+partaf+partag+partah+partai+partaj+partak+partal CentOS5.2.x86.vhd
(all of that should be on one line)
x64: copy /B partaa+partab+partac+partad+partae+partaf+partag+partah+partai+partaj+partak+partal CentOS5.2.x64.vhd
(all of that should be on one line)
You should end up with a single .vhd file with the following size:
x86: 16173279232 bytes
x64: 18433592832 bytes
Then just build a virtual machine around each one with the .vhd file as the IDE hard disk, and with a Network Adapter (not a Legacy Network Adapter) in it.
Remember that the root password is the word “password” (without the quotes).
You will need to edit these files
- /etc/hosts
- /etc/resolv.conf
- /etc/sysconfig/network
and then run the command “setup” to configure the IP address and so on of the “seth0” network device. Then just reboot and you’re away.
If you need a graphical interface with a mouse and such, then change the “3” to a “5” in the “default” line in /etc/inittab and reboot. But I would advise leaving it in text-only mode.
Speeding Up Ext3 Filesystems
02/12/08 15:22
There are a few things you can do to speed up the ext3 filesystem, and when combined they make a lot of difference!
Firstly, you probably don’t need to store the “last accessed” time of every file and/or every directory, so add “noatime,nodiratime” to the mount options in /etc/fstab (add a comma then that text straight after the word “defaults” in the relevant line of /etc/fstab).
Furthermore, you can optimise the caching of data in the filesystem by adding “data=writeback” to the mount options in /etc/fstab. This is pretty safe as long as your system isn’t very busy and liable to lose power without warning. The only downside is that should it lose power while writing to the disk, a few files may end up with slightly old content in them.
The last one is a little more complicated, but well worth doing. You can change the directories to be B-trees instead of lists, which are a lot faster if you have many files in each directory. Say your filesystem is mounted off /dev/sdb1, for example.
Firstly, you probably don’t need to store the “last accessed” time of every file and/or every directory, so add “noatime,nodiratime” to the mount options in /etc/fstab (add a comma then that text straight after the word “defaults” in the relevant line of /etc/fstab).
Furthermore, you can optimise the caching of data in the filesystem by adding “data=writeback” to the mount options in /etc/fstab. This is pretty safe as long as your system isn’t very busy and liable to lose power without warning. The only downside is that should it lose power while writing to the disk, a few files may end up with slightly old content in them.
The last one is a little more complicated, but well worth doing. You can change the directories to be B-trees instead of lists, which are a lot faster if you have many files in each directory. Say your filesystem is mounted off /dev/sdb1, for example.
- Unmount the filesystem, having stopped all processes that are using it, with “umount /dev/sdb1”.
- Change the directory indexing with “tune2fs -O dir_index /dev/sdb1”.
- Re-build all the existing directories with “e2fsck -D /dev/sdb1”.
- Reboot, or else remount the filesystem and start the processes back up. Rebooting is simpler
- That should make your filesystem run a whole lot faster!
Hyper-V Linux Integration Components RC2 Download
28/07/08 15:44
To make it easy to find, the RC2 of the Microsoft Windows Server 2008 Hyper-V Linux Integration Components can be found here: Linux-IC-RC2.
Speeding Up ext3 Filesystems
22/07/08 12:13
There are a few things you can do to speed up the operation of ext3 filesystems, by changing some of the ways in which it behaves by default.
This is all for /dev/sda1, change it to suit the partition you are working with. Make sure you have an ext3 filesystem on there already, and ensure you have unmounted the filesystem first.
umount /dev/sda1
tune2fs -Ohas_journal -o journal_data_writeback /dev/sda1
tune2fs -O dir_index /dev/sda1
e2fsck -D /dev/sda1
Add ",noatime,nodiratime" to the list of options in the relevant line in /etc/fstab.
mount /dev/sda1
This is all for /dev/sda1, change it to suit the partition you are working with. Make sure you have an ext3 filesystem on there already, and ensure you have unmounted the filesystem first.
umount /dev/sda1
tune2fs -Ohas_journal -o journal_data_writeback /dev/sda1
tune2fs -O dir_index /dev/sda1
e2fsck -D /dev/sda1
Add ",noatime,nodiratime" to the list of options in the relevant line in /etc/fstab.
mount /dev/sda1
Extracting OLE Objects from Word Documents
12/06/08 18:12
Many people have asked me how to extract the file embedded inside an OLE object that has been inserted into a Microsoft Word document, or similar.
I reverse-engineered the file format, it’s very simple. Not this code doesn’t always appear to work, but it gets 95% of them out.
Use it at your own peril. Please credit me (Julian Field jules@jules.fm) where/when/if you use this code or any derivative of it, including translations into other languages.
$byte = "";
$buffer = "";
#$infh = new FileHandle;
#sysopen $infh, "$explodeinto/$inname", O_RDONLY;
Open the infh filehandle with the "inname" file containing the OLE object.
sysseek $infh, 6, SEEK_SET; # Skip 1st 6 bytes
Skip the first 6 bytes, these appear to be useless
$outname = "";
$finished = 0;
$length = 0;
until ($byte eq "\0" || $finished || $length>1000) {
# Read a C-string into $outname
sysread($infh, $byte, 1) or $finished = 1;
$outname .= $byte;
$length++;
}
Read a null-terminated string of bytes,
this becomes the output filename.
next OLEFILE if $length>1000; # Bail out if it went wrong
If the filename was way too long, this is probably corrupt.
$finished = 0;
$byte = 1;
$length = 0;
until ($byte eq "\0" || $finished || $length>1000) { # Throw away a C-string
sysread($infh, $byte, 1) or $finished = 1;
$length++;
}
Throw away the next null-terminated string of bytes.
next OLEFILE if $length>1000; # Bail out if it went wrong
If the string was way too long, this is probably corrupt.
sysseek $infh, 4, Fcntl::SEEK_CUR or next OLEFILE; # Skip next 4 bytes
Skip the next 4 bytes of the file.
sysread $infh, $number, 4 or next OLEFILE;
$number = unpack 'V', $number;
Read the next 4 bytes into a 4-byte int called "$number".
#print STDERR "Skipping $number bytes of header filename\n";
if ($number>0 && $number<1_000_000) {
sysseek $infh, $number, Fcntl::SEEK_CUR; # Skip the next bit of header (C-string)
} else {
next OLEFILE;
}
If the number $number was a reasonable size,
skip that many bytes of the file.
sysread $infh, $number, 4 or next OLEFILE;
$number = unpack 'V', $number;
Read the next 4 bytes in a 4-byte int called "$number".
This is the length of the real embedded file we want to extract.
#print STDERR "Reading $number bytes of file data\n";
sysread $infh, $buffer, $number
if $number>0 && $number < $size; # Sanity check
Read the $number number of bytes into memory into a chunk
of memory allocated which is at least $number bytes long.
Do a sanity check that the number of bytes we have asked it to read
is less than the total length of the input file.
$outfh = new FileHandle;
$outsafe = $this->MakeNameSafe($outname, $explodeinto);
sysopen $outfh, "$explodeinto/$outsafe", (O_CREAT | O_WRONLY)
or next OLEFILE;
Create an output file with a filename which is a sanitised safe
version of the filename we read at the top of this bit of code.
if ($number>0 && $number<1_000_000_000) { # Number must be reasonable!
syswrite $outfh, $buffer, $number or next OLEFILE;
}
close $outfh;
If the output file is less than 1Gbyte long, write out the data we just read.
This creates the file containing the embedded file we wanted to extract.
Then close that output file.
I reverse-engineered the file format, it’s very simple. Not this code doesn’t always appear to work, but it gets 95% of them out.
Use it at your own peril. Please credit me (Julian Field jules@jules.fm) where/when/if you use this code or any derivative of it, including translations into other languages.
$byte = "";
$buffer = "";
#$infh = new FileHandle;
#sysopen $infh, "$explodeinto/$inname", O_RDONLY;
Open the infh filehandle with the "inname" file containing the OLE object.
sysseek $infh, 6, SEEK_SET; # Skip 1st 6 bytes
Skip the first 6 bytes, these appear to be useless
$outname = "";
$finished = 0;
$length = 0;
until ($byte eq "\0" || $finished || $length>1000) {
# Read a C-string into $outname
sysread($infh, $byte, 1) or $finished = 1;
$outname .= $byte;
$length++;
}
Read a null-terminated string of bytes,
this becomes the output filename.
next OLEFILE if $length>1000; # Bail out if it went wrong
If the filename was way too long, this is probably corrupt.
$finished = 0;
$byte = 1;
$length = 0;
until ($byte eq "\0" || $finished || $length>1000) { # Throw away a C-string
sysread($infh, $byte, 1) or $finished = 1;
$length++;
}
Throw away the next null-terminated string of bytes.
next OLEFILE if $length>1000; # Bail out if it went wrong
If the string was way too long, this is probably corrupt.
sysseek $infh, 4, Fcntl::SEEK_CUR or next OLEFILE; # Skip next 4 bytes
Skip the next 4 bytes of the file.
sysread $infh, $number, 4 or next OLEFILE;
$number = unpack 'V', $number;
Read the next 4 bytes into a 4-byte int called "$number".
#print STDERR "Skipping $number bytes of header filename\n";
if ($number>0 && $number<1_000_000) {
sysseek $infh, $number, Fcntl::SEEK_CUR; # Skip the next bit of header (C-string)
} else {
next OLEFILE;
}
If the number $number was a reasonable size,
skip that many bytes of the file.
sysread $infh, $number, 4 or next OLEFILE;
$number = unpack 'V', $number;
Read the next 4 bytes in a 4-byte int called "$number".
This is the length of the real embedded file we want to extract.
#print STDERR "Reading $number bytes of file data\n";
sysread $infh, $buffer, $number
if $number>0 && $number < $size; # Sanity check
Read the $number number of bytes into memory into a chunk
of memory allocated which is at least $number bytes long.
Do a sanity check that the number of bytes we have asked it to read
is less than the total length of the input file.
$outfh = new FileHandle;
$outsafe = $this->MakeNameSafe($outname, $explodeinto);
sysopen $outfh, "$explodeinto/$outsafe", (O_CREAT | O_WRONLY)
or next OLEFILE;
Create an output file with a filename which is a sanitised safe
version of the filename we read at the top of this bit of code.
if ($number>0 && $number<1_000_000_000) { # Number must be reasonable!
syswrite $outfh, $buffer, $number or next OLEFILE;
}
close $outfh;
If the output file is less than 1Gbyte long, write out the data we just read.
This creates the file containing the embedded file we wanted to extract.
Then close that output file.
RedHat 5.2 yum update errors
04/06/08 17:21
If you install RedHat 5.2 and try to do a “yum update” command, you get loads of errors about libxslt and libvorbis.
To solve these errors, type the following commands before you do a “yum update”.
# rpm -e --nodeps --allmatches libvorbis
# rpm -e --nodeps --allmatches libxslt-python libxslt-devel
# rpm -e --nodeps --allmatches libxslt
# rpm -e --nodeps --allmatches libvorbis-devel
# yum -y install libvorbis libxslt
Then you will find that “yum update” works as expected.
To solve these errors, type the following commands before you do a “yum update”.
# rpm -e --nodeps --allmatches libvorbis
# rpm -e --nodeps --allmatches libxslt-python libxslt-devel
# rpm -e --nodeps --allmatches libxslt
# rpm -e --nodeps --allmatches libvorbis-devel
# yum -y install libvorbis libxslt
Then you will find that “yum update” works as expected.