Difference between revisions of "UKCA Chemistry and Aerosol Tutorials: Configuring X2Go for AWS"

From UKCA
 
(38 intermediate revisions by the same user not shown)
Line 1: Line 1:
[[UKCA Chemistry and Aerosol Tutorials at vn11.8]]
+
[[UKCA Chemistry and Aerosol Tutorials at UMvn13.0]]
   
To use the UKCA Tutorials on the JASMIN Unmanaged Cloud you first need to download and install X2Go client. This is available for Windows, macOS, and GNU/Linux.
+
''For instructions on using the VMs on the JASMIN Unmanaged Cloud please see [[UKCA Chemistry and Aerosol Tutorials: Configuring X2Go for the JASMIN UMC|here]].''
  +
  +
{| class="wikitable" border = "1px"
 
|-
  +
| Difficulty || '''<span style="color:green">EASY</span>'''
 
|-
  +
| Time to Complete || '''Under 1 hour'''
 
|-
  +
| Video instructions || '''https://youtu.be/_JADT-XKHQY'''
  +
|}
  +
  +
To use the UKCA Tutorials on AWS you first need to download and install X2Go client. This is available for Windows, macOS, and GNU/Linux.
   
 
* '''https://wiki.x2go.org/doku.php/doc:installation:x2goclient'''
 
* '''https://wiki.x2go.org/doku.php/doc:installation:x2goclient'''
   
You will be provided with connection details to your personal VM and the required SSH key file that you will need.
 
   
 
You will be provided with connection details to your personal VM and the required SSH key file that you will need.
== SSH key files ==
 
  +
  +
'''Note''' that the username on these VMs will be '''ubuntu''' and you will be provided a unique IP address to use to connect to the VM (labelled as '''YOUR.ASSIGNED.IP.ADDRESS''' below).
  +
  +
If you are unable to use X2Go, you can also connect via a '''[[#Connecting via a Terminal (GNU/Linux & macOS)|Terminal]]''' on GNU/Linux & macOS, and via '''[[#Connecting via a MobaXTerm (Windows)|MobaXTerm]]''' on Windows.
  +
 
== SSH key file ==
   
Before you set up X2Go, you need to be sure that your SSH-key files are in the correct format. There will be two text files, which look like:
+
Before you set up X2Go, you need to be sure that your SSH-key file is in the correct format. There will be a single text file, which look like:
* '''<tt>id_rsa_ukcatrXX</tt>''': This is your '''private key''' and you should not share this file with anyone. '''It is this file that you need to use to connect using X2Go.''' It does not have a file extension. It will have a large block of random text in between the following lines:
+
* '''<tt>ukca_key_trXX.pem</tt>''': This is your '''private key''' and you should not share this file with anyone. '''It is this file that you need to use to connect using X2Go, MobaXTerm, or SSH.''' It will have a large block of random text in between the following lines:
 
-----BEGIN OPENSSH PRIVATE KEY-----
 
-----BEGIN OPENSSH PRIVATE KEY-----
 
 
 
-----END OPENSSH PRIVATE KEY-----
 
-----END OPENSSH PRIVATE KEY-----
* '''<tt>id_rsa_ukcatrXX.pub</tt>''': This is your '''public key''', and has the .pub file extension. It contains a single line beginning with <tt>ssh-rsa</tt> followed by a long string of random numbers and letters.
 
   
 
== Setting up X2Go ==
 
== Setting up X2Go ==
Line 26: Line 41:
 
!style="text-align:top;"|Setting
 
!style="text-align:top;"|Setting
 
|-
 
|-
| Session name || ''e.g.'' ukcatrXX@ukca-vmXX
+
| Session name || ''e.g.'' ubuntu@ukca_vm_trXX
 
|-
 
|-
  +
| Host || '''YOUR.ASSIGNED.IP.ADDRESS''' ''(e.g. 192.171.139.44)''
| Host || ukca-vmXX
 
 
|-
 
|-
| Login || ukcatrXX
+
| Login || ubuntu
 
|-
 
|-
| Use RSA/DSA key for ssh connection || ''The path to your'' id_rsa_ukcatrXX '' key file (navigate via button)''
+
| Use RSA/DSA key for ssh connection || ''The path to your'' <tt>ukca_key_trXX.pem</tt> '' key file (navigate via button)''
|-
 
| Use Proxy server for SSH connection || ''Tick checkbox''
 
|-
 
| Proxy server – Host || 192.171.139.44
 
|-
 
| Proxy server – same login as on X2Go server || ''Tick checkbox''
 
|-
 
| Proxy server – same password as on X2Go server || ''Tick checkbox''
 
|-
 
| Proxy server – RSA/DSA key || ''The path to your'' id_rsa_ukcatrXX '' key file (navigate via button)''
 
 
|-
 
|-
 
| Session type || ''Select'' LXDE ''from drop-down menu''
 
| Session type || ''Select'' LXDE ''from drop-down menu''
Line 51: Line 56:
 
When you have completed everything, press '''OK'''. The first time you connect you will be asked if you which to accept the host key. Click '''yes'''. Once you have set the preferences you should get a grey box representing your session, e.g.
 
When you have completed everything, press '''OK'''. The first time you connect you will be asked if you which to accept the host key. Click '''yes'''. Once you have set the preferences you should get a grey box representing your session, e.g.
   
[[Image:X2Go_session_box.png|200px]]
+
[[Image:X2Go_session_box_AWS.png|200px]]
   
You should double-click at the top of the box (i.e. over the session name or on the icon) to start the connection. If you have closed a session down but left X2Go running you may instead see a grey box with the login field as <tt>ukcatrXX</tt> and the password field blank - to connect in this case just press '''OK'''.
+
You should '''double-click at the top of the box''' (i.e. over the session name or on the icon) '''to start the connection'''. If you have closed a session down but left X2Go running you may instead see a grey box with the login field as <tt>ubuntu</tt> and the password field blank - to connect in this case just press '''OK'''.
   
 
When you connect to your VM using X2Go you will be presented with an open Linux terminal which is asking for your MOSRS password. Once you have entered it, the first time you login you will also be asked for your MOSRS username. Subsequently, you will just be asked for your MOSRS password.
 
When you connect to your VM using X2Go you will be presented with an open Linux terminal which is asking for your MOSRS password. Once you have entered it, the first time you login you will also be asked for your MOSRS username. Subsequently, you will just be asked for your MOSRS password.
Line 60: Line 65:
   
 
<gallery widths=450px heights=600px>
 
<gallery widths=450px heights=600px>
Image:X2Go_settings.png|Example X2Go settings panel
+
Image:X2Go_settings_AWS.png|Example X2Go settings panel
 
Image:X2Go_terminal_login.png|X2Go connection to a VM, showing the terminal connecting to MOSRS for the first time.
 
Image:X2Go_terminal_login.png|X2Go connection to a VM, showing the terminal connecting to MOSRS for the first time.
 
</gallery>
 
</gallery>
  +
  +
=== Troubleshooting ===
  +
  +
You may find that you cannot copy and paste between your personal computer and the VM.
  +
  +
* On Windows you should follow [https://stackoverflow.com/questions/66872307/x2go-clipboard-copy-paste-stops-working-from-windows-10-to-linux-centos-and-v these instructions]:
  +
** Open X2GO Client application
  +
** Click "Options" menu --> Settings
  +
** Select "X. Org Server Settings" tab
  +
** Select "do not use primary clipboard" (make sure its checked)
  +
** Click OK
  +
** Restart your X2Go client
  +
* On macOS you will need to follow [https://wiki.x2go.org/doku.php/doc:installation:x2goclient#os_x these instructions]:
  +
** In a Terminal, enter the following command
  +
echo "*VT100.translations: #override Meta <KeyPress> V: insert-selection(PRIMARY, CUT_BUFFER0) \n" > ~/.Xdefaults
   
 
== Virtual Machine Environment ==
 
== Virtual Machine Environment ==
   
You will be using X2Go to connect to a Virtual Machine (VM) hosted on the JASMIN Unmanaged Cloud, which has been specially set-up to be able to run the UKCA Tutorials. All the relevant files and settings have already been installed, all you need to do is to enter your MOSRS credentials when you first connect, and then cache your [[#MOSRS_Password|MOSRS password]] once a day or in a new terminal as required.
+
You will be using X2Go to connect to a Virtual Machine (VM) hosted on the AWS cloud, which has been specially set-up to be able to run the UKCA Tutorials. All the relevant files and settings have already been installed, all you need to do is to enter your MOSRS credentials when you first connect, and then cache your [[#MOSRS_Password|MOSRS password]] once a day or in a new terminal as required.
   
 
The VM you are connecting to uses the Ubuntu 18.04 operating system, with the LXDE desktop. While you have access to the full desktop, you should be comfortable using a GNU/Linux operating system, including using and navigating around using a '''terminal'''. You should also be familiar with programming, ideally '''Fortran'''.
 
The VM you are connecting to uses the Ubuntu 18.04 operating system, with the LXDE desktop. While you have access to the full desktop, you should be comfortable using a GNU/Linux operating system, including using and navigating around using a '''terminal'''. You should also be familiar with programming, ideally '''Fortran'''.
Line 82: Line 102:
 
== Connecting via a Terminal (GNU/Linux & macOS) ==
 
== Connecting via a Terminal (GNU/Linux & macOS) ==
   
It is also possible to connect via a terminal with an X11 connection ([https://www.xquartz.org/ XQuartz] would be required when using macOS). The command to use in this case would be
+
It is also possible to connect via a terminal with an X11 connection ([https://www.xquartz.org/ XQuartz] is also required when using macOS). The command to use in this case would be
  +
ssh -X -i /path/to/ukca_key_trXX.pem ubuntu@'''YOUR.ASSIGNED.IP.ADDRESS'''
ssh -Y -o ProxyCommand="ssh -W %h:%p -q -i /path/to/id_rsa_ukcatrXX ukcatrXX@192.171.139.44" -i /path/to/id_rsa_ukcatrXX ukcatrXX@ukca-vmXX
 
where <tt>/path/to/id_rsa_ukcatrXX</tt> is the full path to your SSH private key file.
+
where <tt>/path/to/ukca_key_trXX.pem</tt> is the full path to your SSH private key file.
   
 
It is also possible to define a <tt>.ssh/config</tt> file entry with the necessary information if desired:
 
It is also possible to define a <tt>.ssh/config</tt> file entry with the necessary information if desired:
Host ukca-vmXX
+
Host ukca_vm_trXX
Hostname ukca-vmXX
+
Hostname '''YOUR.ASSIGNED.IP.ADDRESS'''
User ukcatrXX
+
User ubuntu
IdentityFile /path/to/id_rsa_ukcatrX
+
IdentityFile /path/to/ukca_key_trXX.pem
  +
ProxyCommand ssh -W %h:%p -q -i /path/to/id_rsa_ukcatrXX ukcatrXX@192.171.139.44
 
 
so that you could then just connect using the command
 
so that you could then just connect using the command
ssh -Y ukca-vmXX
+
ssh -X ukca_vm_trXX
   
 
When you connect via a terminal you won't have a full desktop environment - you will need to navigate around the VM from the command line in the usual way.
 
When you connect via a terminal you won't have a full desktop environment - you will need to navigate around the VM from the command line in the usual way.
  +
  +
'''If you have any issues authenticating to MOSRS or using Rose you could also try using <tt>-Y</tt> instead of <tt>-X</tt>.'''
   
 
== Connecting via a MobaXTerm (Windows) ==
 
== Connecting via a MobaXTerm (Windows) ==
Line 103: Line 125:
 
* '''https://mobaxterm.mobatek.net/'''
 
* '''https://mobaxterm.mobatek.net/'''
   
  +
To get started, you will to click '''Session''' and then '''SSH''' to create a new SSH session with the following equivalent settings:
+
To get started, you should click on '''Session''' and then '''SSH''' to create a new SSH session with the following equivalent settings:
   
 
{| class="wikitable" border = "1px"
 
{| class="wikitable" border = "1px"
Line 111: Line 134:
 
! scope="row" colspan="2" style="text-align:left;"| Basic SSH settings (top row)
 
! scope="row" colspan="2" style="text-align:left;"| Basic SSH settings (top row)
 
|-
 
|-
| Remote host || ukca-vmXX
+
| Remote host || '''YOUR.ASSIGNED.IP.ADDRESS''' ''(e.g. 192.171.139.44)''
 
|-
 
|-
 
| Specify username || ''Tick checkbox''
 
| Specify username || ''Tick checkbox''
 
|-
 
|-
| || ukcatrXX
+
| || ubuntu
 
|-
 
|-
 
! scope="row" colspan="2" style="text-align:left;"| Advanced SSH settings Tab
 
! scope="row" colspan="2" style="text-align:left;"| Advanced SSH settings Tab
Line 123: Line 146:
 
| Use private key || ''Tick checkbox''
 
| Use private key || ''Tick checkbox''
 
|-
 
|-
| || ''The path to your'' id_rsa_ukcatrXX '' key file (navigate via button)''
+
| || ''The path to your'' <tt>ukca_key_trXX.pem</tt> '' key file (navigate via button)''
 
|-
 
|-
! scope="row" colspan="2" style="text-align:left;"| Network settings Tab
+
! scope="row" colspan="2" style="text-align:left;"| Advanced SSH settings Tab - optional choices
 
|-
 
|-
  +
| Remote environment (drop-down menu) || ''Interactive shell'' to connect via the MobaXTerm terminal
| Connect through SSH gateway (jump host) || ''Tick checkbox''
 
|-
+
|-
  +
| || ''LXDE'' to connect with a grahpical environment
| Gateway SSH server || 192.171.139.44
 
|-
 
| User || ukcatrXX
 
|-
 
| Use private key || ''Tick checkbox''
 
|-
 
| || ''The path to your'' id_rsa_ukcatrXX '' key file (navigate via button)''
 
 
|}
 
|}
   
 
Once you have connected you will need to navigate around and open files using terminal commands.
 
Once you have connected you will need to navigate around and open files using terminal commands.
   
<gallery widths=450px heights=400px>
+
<gallery widths=450px heights=300px>
Image:MobaXTerm_advanced.png|Example MobaXTerm advanced SSH settings panel.
+
Image:MobaXTerm_advanced_AWS.png|Example MobaXTerm advanced SSH settings panel.
  +
Image:MobaXTerm_LXDE.png|Settings for connecting to the LXDE desktop environment.
Image:MobaXTerm_network.png|Example MobaXTerm network settings panel.
 
 
</gallery>
 
</gallery>
  +
  +
== Copying data back ==
  +
  +
If you are using a GNU/Linux or macOS system and you have set up the <tt>.ssh/config</tt> file as [[#Connecting_via_a_Terminal_.28GNU.2FLinux_.26_macOS.29|above]], you can pull data from the VMs using the following recommended '''<tt>rsync</tt>''' command, e.g.
  +
  +
rsync -auvv ukca_vm_trXX:foo.py .
  +
  +
where <tt>foo.py</tt> is a file in your <tt>home/</tt> directory to your current working directory on your local system. To copy from other directories you could use the command, e.g.
  +
  +
rsync -auvv ukca_vm_trXX:/home/ubuntu/cylc-run/SUITE-ID/work/1/atmos/atmosa.pa19810901_00 .
  +
  +
The following longer command also works:
  +
  +
rsync -auvv -e "ssh -i /path/to/ukca_key_trXX.pem" ubuntu@'''YOUR.ASSIGNED.IP.ADDRESS''':foo.py .
  +
  +
where <tt>/path/to/ukca_key_trXX.pem</tt> above means the full path to your SSH private key file, e.g. <tt>/Users/luke/ukca_key_tr01.pem</tt> etc.
  +
  +
You can also copy directories in a similar way, e.g.
  +
  +
rsync -auvv -e "ssh -i /path/to/ukca_key_trXX.pem" ubuntu@'''YOUR.ASSIGNED.IP.ADDRESS''':/home/ubuntu/roses .
  +
  +
You could also use Firefox to log-in to your e-mail and e-mail the files to yourself, or log-in to a Cloud service, such as Google Drive and Dropbox, and upload the files there directly.
  +
  +
Make sure you '''<tt>fcm commit</tt>''' your suite(s) and branch(es) - this will allow you to use these later on a personal VM, although note that the paths within the suites will need to be changed later, as the username will be <tt>vagrant</tt> when using the [https://github.com/metomi/metomi-vms Met Office Virtual Machine].
  +
  +
== Further Information ==
  +
  +
For further information on how these VMs are set-up and run on AWS, please see the following [https://github.com/theabro/aws-ukca GitHub repository] containing all the scripts and instructions used to generate the environment.
   
 
----
 
----
''Written by [[User:Nla27 | Luke Abraham]] 2021.
+
''Written by [[User:Nla27 | Luke Abraham]] 2022.

Latest revision as of 19:08, 29 January 2024

UKCA Chemistry and Aerosol Tutorials at UMvn13.0

For instructions on using the VMs on the JASMIN Unmanaged Cloud please see here.

Difficulty EASY
Time to Complete Under 1 hour
Video instructions https://youtu.be/_JADT-XKHQY

To use the UKCA Tutorials on AWS you first need to download and install X2Go client. This is available for Windows, macOS, and GNU/Linux.


You will be provided with connection details to your personal VM and the required SSH key file that you will need.

Note that the username on these VMs will be ubuntu and you will be provided a unique IP address to use to connect to the VM (labelled as YOUR.ASSIGNED.IP.ADDRESS below).

If you are unable to use X2Go, you can also connect via a Terminal on GNU/Linux & macOS, and via MobaXTerm on Windows.

SSH key file

Before you set up X2Go, you need to be sure that your SSH-key file is in the correct format. There will be a single text file, which look like:

  • ukca_key_trXX.pem: This is your private key and you should not share this file with anyone. It is this file that you need to use to connect using X2Go, MobaXTerm, or SSH. It will have a large block of random text in between the following lines:
-----BEGIN OPENSSH PRIVATE KEY-----

-----END OPENSSH PRIVATE KEY-----

Setting up X2Go

When first starting X2Go, you should be presented with a new session dialog box to complete. If this doesn't appear, press the "New Session" button (which looks like a piece of paper with a star on it). Once this is open, fill in the details below.

When setting up an X2Go session, the exact value of “session name” is not important. The key details are the settings under “server” and “proxy server”. You will also need to set the “session type” to LXDE.

Option Setting
Session name e.g. ubuntu@ukca_vm_trXX
Host YOUR.ASSIGNED.IP.ADDRESS (e.g. 192.171.139.44)
Login ubuntu
Use RSA/DSA key for ssh connection The path to your ukca_key_trXX.pem key file (navigate via button)
Session type Select LXDE from drop-down menu

Note: On macOS you may need to maximise the Session Preferences window (using the green + marker in the top left corner) to allow you to set the session type and press the OK button, as this panel is quite long and these buttons will then become lost below the screen.

When you have completed everything, press OK. The first time you connect you will be asked if you which to accept the host key. Click yes. Once you have set the preferences you should get a grey box representing your session, e.g.

X2Go session box AWS.png

You should double-click at the top of the box (i.e. over the session name or on the icon) to start the connection. If you have closed a session down but left X2Go running you may instead see a grey box with the login field as ubuntu and the password field blank - to connect in this case just press OK.

When you connect to your VM using X2Go you will be presented with an open Linux terminal which is asking for your MOSRS password. Once you have entered it, the first time you login you will also be asked for your MOSRS username. Subsequently, you will just be asked for your MOSRS password.

When you have finished for the day you can close the connection by closing the X2Go window using the X in the corner. When you reconnect you will find the windows exactly as you left them. You can also resize the X2Go window using the maximize button or by dragging the corners with your mouse.

Troubleshooting

You may find that you cannot copy and paste between your personal computer and the VM.

  • On Windows you should follow these instructions:
    • Open X2GO Client application
    • Click "Options" menu --> Settings
    • Select "X. Org Server Settings" tab
    • Select "do not use primary clipboard" (make sure its checked)
    • Click OK
    • Restart your X2Go client
  • On macOS you will need to follow these instructions:
    • In a Terminal, enter the following command
echo "*VT100.translations: #override Meta <KeyPress> V: insert-selection(PRIMARY, CUT_BUFFER0) \n" > ~/.Xdefaults

Virtual Machine Environment

You will be using X2Go to connect to a Virtual Machine (VM) hosted on the AWS cloud, which has been specially set-up to be able to run the UKCA Tutorials. All the relevant files and settings have already been installed, all you need to do is to enter your MOSRS credentials when you first connect, and then cache your MOSRS password once a day or in a new terminal as required.

The VM you are connecting to uses the Ubuntu 18.04 operating system, with the LXDE desktop. While you have access to the full desktop, you should be comfortable using a GNU/Linux operating system, including using and navigating around using a terminal. You should also be familiar with programming, ideally Fortran.

MOSRS Password

You will need to enter your MOSRS password each day, and for each terminal that you open on the VM when using X2Go that you wish to commit changes or run suites from.

If you leave your terminal open overnight, in the morning you should use the command

mosrs-cache-password

to cache your password for the day.

If you do not do this each morning you may get some strange errors when committing changes or when running suites.

Connecting via a Terminal (GNU/Linux & macOS)

It is also possible to connect via a terminal with an X11 connection (XQuartz is also required when using macOS). The command to use in this case would be

ssh -X -i /path/to/ukca_key_trXX.pem ubuntu@YOUR.ASSIGNED.IP.ADDRESS

where /path/to/ukca_key_trXX.pem is the full path to your SSH private key file.

It is also possible to define a .ssh/config file entry with the necessary information if desired:

Host ukca_vm_trXX
       Hostname YOUR.ASSIGNED.IP.ADDRESS
       User ubuntu
       IdentityFile /path/to/ukca_key_trXX.pem

so that you could then just connect using the command

ssh -X ukca_vm_trXX

When you connect via a terminal you won't have a full desktop environment - you will need to navigate around the VM from the command line in the usual way.

If you have any issues authenticating to MOSRS or using Rose you could also try using -Y instead of -X.

Connecting via a MobaXTerm (Windows)

If you are unable to use X2Go, it is also possible to connect from Windows via a Terminal client such as MobaXTerm


To get started, you should click on Session and then SSH to create a new SSH session with the following equivalent settings:

Option Setting
Basic SSH settings (top row)
Remote host YOUR.ASSIGNED.IP.ADDRESS (e.g. 192.171.139.44)
Specify username Tick checkbox
ubuntu
Advanced SSH settings Tab
X11 forwarding Ensure that the checkbox is ticked (default)
Use private key Tick checkbox
The path to your ukca_key_trXX.pem key file (navigate via button)
Advanced SSH settings Tab - optional choices
Remote environment (drop-down menu) Interactive shell to connect via the MobaXTerm terminal
LXDE to connect with a grahpical environment

Once you have connected you will need to navigate around and open files using terminal commands.

Copying data back

If you are using a GNU/Linux or macOS system and you have set up the .ssh/config file as above, you can pull data from the VMs using the following recommended rsync command, e.g.

rsync -auvv ukca_vm_trXX:foo.py .

where foo.py is a file in your home/ directory to your current working directory on your local system. To copy from other directories you could use the command, e.g.

rsync -auvv ukca_vm_trXX:/home/ubuntu/cylc-run/SUITE-ID/work/1/atmos/atmosa.pa19810901_00 .

The following longer command also works:

rsync -auvv -e "ssh -i /path/to/ukca_key_trXX.pem" ubuntu@YOUR.ASSIGNED.IP.ADDRESS:foo.py .

where /path/to/ukca_key_trXX.pem above means the full path to your SSH private key file, e.g. /Users/luke/ukca_key_tr01.pem etc.

You can also copy directories in a similar way, e.g.

rsync -auvv -e "ssh -i /path/to/ukca_key_trXX.pem" ubuntu@YOUR.ASSIGNED.IP.ADDRESS:/home/ubuntu/roses .

You could also use Firefox to log-in to your e-mail and e-mail the files to yourself, or log-in to a Cloud service, such as Google Drive and Dropbox, and upload the files there directly.

Make sure you fcm commit your suite(s) and branch(es) - this will allow you to use these later on a personal VM, although note that the paths within the suites will need to be changed later, as the username will be vagrant when using the Met Office Virtual Machine.

Further Information

For further information on how these VMs are set-up and run on AWS, please see the following GitHub repository containing all the scripts and instructions used to generate the environment.


Written by Luke Abraham 2022.