Note: The default ITS GitLab runner is a shared resource and is subject to slowdowns during heavy usage.
You can run your own GitLab runner that is dedicated just to your group if you need to avoid processing delays.

Commit f564dbee authored by Qusai Al Shidi's avatar Qusai Al Shidi 💬
Browse files

Merge branch 'master' into imf-f107

parents 339c1911 d4a5c35f
......@@ -14,6 +14,8 @@ git checkout -b my_new_feature
and start commiting there to test and work on. We will follow a [trunk-based development model](https://youtu.be/ykZbBD-CmP8). This means we will rapidly merge branches once you have something stable and continue with master branch. So make sure to push something stable instead of being in your feature branch for too long. I will try to address your pull request ASAP.
[![Trunk-based Development](http://img.youtube.com/vi/ykZbBD-CmP8/0.jpg)](https://www.youtube.com/watch?v=ykZbBD-CmP8 "Trunk-based Development")
Coding Style: PEP 8
-------------------
......
# Table of Contents
* [swmfpy](#.swmfpy)
* [swmfpy.tools](#.swmfpy.tools)
* [swmfpy.io](#.swmfpy.io)
* [read\_wdc\_ae](#.swmfpy.io.read_wdc_ae)
* [read\_wdc\_asy\_sym](#.swmfpy.io.read_wdc_asy_sym)
* [read\_gm\_log](#.swmfpy.io.read_gm_log)
* [swmfpy.tools](#.swmfpy.tools)
* [swmfpy.paramin](#.swmfpy.paramin)
* [replace\_command](#.swmfpy.paramin.replace_command)
* [read\_command](#.swmfpy.paramin.read_command)
* [swmfpy.web](#.swmfpy.web)
* [get\_omni\_data](#.swmfpy.web.get_omni_data)
* [download\_magnetogram\_hmi](#.swmfpy.web.download_magnetogram_hmi)
* [download\_magnetogram\_adapt](#.swmfpy.web.download_magnetogram_adapt)
* [swmfpy.paramin](#.swmfpy.paramin)
* [replace\_command](#.swmfpy.paramin.replace_command)
* [read\_command](#.swmfpy.paramin.read_command)
<a name=".swmfpy"></a>
## swmfpy
......@@ -20,8 +20,7 @@
A collection of tools to read, write, visualize with the
Space Weather Modeling Framework (SWMF).
Modules
-------
### Modules
These are automatically imported.
......@@ -29,22 +28,25 @@ These are automatically imported.
- `swmfpy.paramin` PARAM.in editing tools.
- `swmfpy.web` Internet data downloading/uploading tools.
Extra Modules
-------------
### Extra Modules
These are not automatically imported. Might have extra dependancies.
*None yet.*
<a name=".swmfpy.tools"></a>
## swmfpy.tools
Tools to be used in swmfpy functions and classes. Some of the functions are
*hidden functions*.
<a name=".swmfpy.io"></a>
## swmfpy.io
Input/Output tools
### Input/Output tools
Here are tools to read and write files relating to SWMF.
TODO: Move pandas dependancy elsewhere.
<a name=".swmfpy.io.read_wdc_ae"></a>
#### read\_wdc\_ae
......@@ -142,109 +144,15 @@ from the GM model log.
plt.plot(geo['times', geo['AL'])
```
<a name=".swmfpy.tools"></a>
## swmfpy.tools
Tools to be used in swmfpy functions and classes. Some of the functions are
*hidden functions*.
<a name=".swmfpy.paramin"></a>
## swmfpy.paramin
These tools are to help script the editing of PARAM.in files.
<a name=".swmfpy.paramin.replace_command"></a>
#### replace\_command
```python
replace_command(parameters, input_file, output_file='PARAM.in')
```
Replace values for the parameters in a PARAM.in file.
Note, if you have repeat commands this will replace all the repeats.
**Arguments**:
- `parameters` _dict_ - Dictionary of strs with format
replace = 'COMMAND': ['value', 'comments', ...]
This is case sensitive.
- `input_file` _str_ - String of PARAM.in file name.
- `output_file` _str_ - (default 'PARAM.in') The output file to write to.
A value of None will not output a file.
**Returns**:
A list of lines of the PARAM.in file that would be outputted.
**Raises**:
- `TypeError` - If a value given couldn't be converted to string.
**Examples**:
```python
change['`SOLARWINDFILE`'] = [['T', 'UseSolarWindFile'],
['new_imf.dat', 'NameSolarWindFile']]
# This will overwrite PARAM.in
swmfpy.paramin.replace('PARAM.in.template', change)
```
<a name=".swmfpy.paramin.read_command"></a>
#### read\_command
```python
read_command(command, paramin_file='PARAM.in', **kwargs)
```
Get parameters of a certain command in PARAM.in file.
This will find the COMMAND and return a list of
values for the parameters.
**Arguments**:
- `command` _str_ - This is the COMMAND you're looking for.
- `paramin_file` _str_ - (default: 'PARAM.in') The file in which you're
looking for the command values.
**kwargs:
- `num_of_values` _int_ - (default: None) Number of values to take from
command.
**Returns**:
- `list` - Values found for the COMMAND in file. Index 0 is
COMMAND and the values follow (1 for first argument...)
**Raises**:
- `ValueError` - When the COMMAND is not found.
**Examples**:
```python
start_time = swmfpy.paramin.read_command('`STARTTIME`')
end_time = swmfpy.paramin.read_command('`ENDTIME`')
print('Starting month is ', start_time[1])
print('Ending month is ', end_time[1])
```
This will treat all following lines as values for the command. To suppress
this, try using the `num_of_values` keyword. This is helpful if your
PARAM.in is comment heavy.
<a name=".swmfpy.web"></a>
## swmfpy.web
Tools to retrieve and send data on the web.
### Tools to download/upload data on the web
Here are a collection of tools to work with data on the internet. Thus,
this module mostly requires an internet connection.
this module mostly requires an internet connection. Which on some
supercomputers would be turned off during a job run. In scripts, make sure to
use these to preprocess before submitting jobs.
<a name=".swmfpy.web.get_omni_data"></a>
#### get\_omni\_data
......@@ -289,7 +197,7 @@ use swmfpy.io.read_omni_data().
#### download\_magnetogram\_hmi
```python
download_magnetogram_hmi(mag_time, hmi_map, **kwargs)
download_magnetogram_hmi(mag_time, hmi_map='hmi.B_720s', **kwargs)
```
Downloads HMI vector magnetogram fits files.
......@@ -402,3 +310,95 @@ pattern: adapt4[0,1]3*yyyymmddhh
download_dir='./mymaps/')
```
<a name=".swmfpy.paramin"></a>
## swmfpy.paramin
### Editing PARAM.in files
These tools are to help script the editing and reading of PARAM.in files.
<a name=".swmfpy.paramin.replace_command"></a>
#### replace\_command
```python
replace_command(parameters, input_file, output_file='PARAM.in')
```
Replace values for the parameters in a PARAM.in file.
Note, if you have repeat commands this will replace all the repeats.
**Arguments**:
- `parameters` _dict_ - Dictionary of strs with format
replace = '#COMMAND': ['value', 'comments', ...]
This is case sensitive.
- `input_file` _str_ - String of PARAM.in file name.
- `output_file` _str_ - (default 'PARAM.in') The output file to write to.
A value of None will not output a file.
**Returns**:
A list of lines of the PARAM.in file that would be outputted.
**Raises**:
- `TypeError` - If a value given couldn't be converted to string.
**Examples**:
```python
change['#SOLARWINDFILE'] = [['T', 'UseSolarWindFile'],
['new_imf.dat', 'NameSolarWindFile']]
# This will overwrite PARAM.in
swmfpy.paramin.replace('PARAM.in.template', change)
```
<a name=".swmfpy.paramin.read_command"></a>
#### read\_command
```python
read_command(command, paramin_file='PARAM.in', **kwargs)
```
Get parameters of a certain command in PARAM.in file.
This will find the #COMMAND and return a list of
values for the parameters.
**Arguments**:
- `command` _str_ - This is the #COMMAND you're looking for.
- `paramin_file` _str_ - (default: 'PARAM.in') The file in which you're
looking for the command values.
**kwargs:
- `num_of_values` _int_ - (default: None) Number of values to take from
command.
**Returns**:
- `list` - Values found for the #COMMAND in file. Index 0 is
#COMMAND and the values follow (1 for first argument...)
**Raises**:
- `ValueError` - When the #COMMAND is not found.
**Examples**:
```python
start_time = swmfpy.paramin.read_command('#STARTTIME')
end_time = swmfpy.paramin.read_command('#ENDTIME')
print('Starting month is ', start_time[1])
print('Ending month is ', end_time[1])
```
This will treat all following lines as values for the command. To suppress
this, try using the `num_of_values` keyword. This is helpful if your
PARAM.in is comment heavy.
![swmfpy logo](share/logo/swmfpy.png "swmfpy")
swmfpy
======
......@@ -5,13 +7,13 @@ A collection of tools to make it easier to work with Python and Space Weather Mo
This is a work in progress.
Usage
-----
*Note*: swmfpy also is part of the SWMF and gets cloned into `SWMF/share/Python`. If you ran `Config.pl -install` while installing SWMF you probably already have it installed in your Python distribution. If you have already done this it is installed on your system. However, if you would like to [develop](CONTRIBUTING.markdown) for swmfpy or have a local copy continue reading.
Installation
------------
Clone into the directory you want to use it.
*Note*: swmfpy also is part of the SWMF and gets cloned into `SWMF/share/Python`. However, if you would like to [develop](CONTRIBUTING.markdown) for swmfpy or have a local copy do the following:
```bash
git clone https://gitlab.umich.edu/swmf_sofware/swmfpy.git /path/to/my/dir
```
......@@ -33,7 +35,9 @@ import swmfpy
Documentation
-------------
Documentation is included as docstrings. Use python's *dir()* and *help()* inbuilt functions to see documentation.
An auto-documented version can be found [here](DOCUMENTATION.markdown).
However, documentation is included as docstrings. Use python's *dir()* and *help()* inbuilt functions to see documentation.
```python
import swmfpy
......@@ -41,10 +45,6 @@ help(swmfpy) # To see list of functions
help(swmfpy.io.read_gm_log) # To see the function documentation
```
However if you would like an auto-documented version (uses exact same text as the help() function output), go [here](DOCUMENTATION.markdown).
*Note*: The autodoc tool has troubles escaping the `#` symbol so examples with `#COMMAND`s in `PARAM.in` files will not show the symbol. Perhaps I can find a fix but the `help()` pages are always better and up to date.
Issues
------
......
#!/usr/bin/env bash
# Script to make the docs for swmfpy.
# Requires pydoc-markdown
# run `pip install pydoc-markdown --user`
# then run this script in the project root directory
pydoc-markdown -v -I. --render-toc -m swmfpy -m swmfpy.web -m swmfpy.io -m swmfpy.paramin -m swmfpy.tools | sed 's/\\043/#/g' - > DOCUMENTATION.markdown
"""A collection of tools to read, write, visualize with the
Space Weather Modeling Framework (SWMF).
Modules
-------
### Modules
These are automatically imported.
......@@ -10,8 +9,7 @@ These are automatically imported.
- `swmfpy.paramin` PARAM.in editing tools.
- `swmfpy.web` Internet data downloading/uploading tools.
Extra Modules
-------------
### Extra Modules
These are not automatically imported. Might have extra dependancies.
......@@ -29,3 +27,47 @@ assert sys.version_info >= (3, 6), "swmfpy requires Python >=3.6. Sorry :(."
from . import paramin
from . import io
from . import web
# This is straight from the format guide on spdf
OMNI_COLS = ('ID for IMF spacecraft',
'ID for SW Plasma spacecraft',
'# of points in IMF averages',
'# of points in Plasma averages',
'Percent interp',
'Timeshift, sec',
'RMS, Timeshift',
'RMS, Phase front normal',
'Time btwn observations, sec',
'Field magnitude average, nT',
'Bx, nT (GSE, GSM)',
'By, nT (GSE)',
'Bz, nT (GSE)',
'By, nT (GSM)',
'Bz, nT (GSM)',
'RMS SD B scalar, nT',
'RMS SD field vector, nT',
'Flow speed, km/s',
'Vx Velocity, km/s, GSE',
'Vy Velocity, km/s, GSE',
'Vz Velocity, km/s, GSE',
'Proton Density, n/cc',
'Temperature, K',
'Flow pressure, nPa',
'Electric field, mV/m',
'Plasma beta',
'Alfven mach number',
'X(s/c), GSE, Re',
'Y(s/c), GSE, Re',
'Z(s/c), GSE, Re',
'BSN location, Xgse, Re',
'BSN location, Ygse, Re',
'BSN location, Zgse, Re',
'AE-index, nT',
'AL-index, nT',
'AU-index, nT',
'SYM/D index, nT',
'SYM/H index, nT',
'ASY/D index, nT',
'ASY/H index, nT',
'PC(N) index',
'Magnetosonic mach number')
"""Input/Output tools
"""### Input/Output tools
Here are tools to read and write files relating to SWMF.
TODO: Move pandas dependancy elsewhere.
"""
import datetime as dt
......
""" These tools are to help script the editing of PARAM.in files.
"""### Editing PARAM.in files
These tools are to help script the editing and reading of PARAM.in files.
"""
__all__ = [
'read_command',
......@@ -18,7 +20,7 @@ def replace_command(parameters, input_file, output_file='PARAM.in'):
Args:
parameters (dict): Dictionary of strs with format
replace = 'COMMAND': ['value', 'comments', ...]
replace = '\043COMMAND': ['value', 'comments', ...]
This is case sensitive.
input_file (str): String of PARAM.in file name.
output_file (str): (default 'PARAM.in') The output file to write to.
......@@ -31,7 +33,7 @@ def replace_command(parameters, input_file, output_file='PARAM.in'):
Examples:
```python
change['#SOLARWINDFILE'] = [['T', 'UseSolarWindFile'],
change['\043SOLARWINDFILE'] = [['T', 'UseSolarWindFile'],
['new_imf.dat', 'NameSolarWindFile']]
# This will overwrite PARAM.in
swmfpy.paramin.replace('PARAM.in.template', change)
......@@ -73,11 +75,11 @@ def replace_command(parameters, input_file, output_file='PARAM.in'):
def read_command(command, paramin_file='PARAM.in', **kwargs):
"""Get parameters of a certain command in PARAM.in file.
This will find the COMMAND and return a list of
This will find the \043COMMAND and return a list of
values for the parameters.
Args:
command (str): This is the COMMAND you're looking for.
command (str): This is the \043COMMAND you're looking for.
paramin_file (str): (default: 'PARAM.in') The file in which you're
looking for the command values.
**kwargs:
......@@ -85,16 +87,16 @@ def read_command(command, paramin_file='PARAM.in', **kwargs):
command.
Returns:
list: Values found for the COMMAND in file. Index 0 is
COMMAND and the values follow (1 for first argument...)
list: Values found for the \043COMMAND in file. Index 0 is
\043COMMAND and the values follow (1 for first argument...)
Raises:
ValueError: When the COMMAND is not found.
ValueError: When the \043COMMAND is not found.
Examples:
```python
start_time = swmfpy.paramin.read_command('#STARTTIME')
end_time = swmfpy.paramin.read_command('#ENDTIME')
start_time = swmfpy.paramin.read_command('\043STARTTIME')
end_time = swmfpy.paramin.read_command('\043ENDTIME')
print('Starting month is ', start_time[1])
print('Ending month is ', end_time[1])
```
......@@ -140,13 +142,13 @@ def read_command(command, paramin_file='PARAM.in', **kwargs):
# HIDDEN FUNCTIONS
def _get_command(line):
"""Returns the 'COMMAND' if on line.
"""Returns the '\043COMMAND' if on line.
Args:
line (str, list, tuple): The line in the PARAM.in file.
Returns:
(str): '#COMMAND' if found and None if not.
(str): '\043COMMAND' if found and None if not.
"""
if isinstance(line, (str, list, tuple)): # Raises type error otherwise
if isinstance(line, str):
......
"""Tools to retrieve and send data on the web.
"""### Tools to download/upload data on the web
Here are a collection of tools to work with data on the internet. Thus,
this module mostly requires an internet connection.
this module mostly requires an internet connection. Which on some
supercomputers would be turned off during a job run. In scripts, make sure to
use these to preprocess before submitting jobs.
"""
__author__ = 'Qusai Al Shidi'
__email__ = 'qusai@umich.edu'
import datetime as dt
import urllib
from .__init__ import OMNI_COLS
from .tools import _import_error_string, _nearest
......@@ -44,55 +47,13 @@ def get_omni_data(time_from, time_to, **kwargs):
from dateutil import rrule
# This is straight from the format guide on spdf
col_names = ('ID for IMF spacecraft',
'ID for SW Plasma spacecraft',
'# of points in IMF averages',
'# of points in Plasma averages',
'Percent interp',
'Timeshift, sec',
'RMS, Timeshift',
'RMS, Phase front normal',
'Time btwn observations, sec',
'Field magnitude average, nT',
'Bx, nT (GSE, GSM)',
'By, nT (GSE)',
'Bz, nT (GSE)',
'By, nT (GSM)',
'Bz, nT (GSM)',
'RMS SD B scalar, nT',
'RMS SD field vector, nT',
'Flow speed, km/s',
'Vx Velocity, km/s, GSE',
'Vy Velocity, km/s, GSE',
'Vz Velocity, km/s, GSE',
'Proton Density, n/cc',
'Temperature, K',
'Flow pressure, nPa',
'Electric field, mV/m',
'Plasma beta',
'Alfven mach number',
'X(s/c), GSE, Re',
'Y(s/c), GSE, Re',
'Z(s/c), GSE, Re',
'BSN location, Xgse, Re',
'BSN location, Ygse, Re',
'BSN location, Zgse, Re',
'AE-index, nT',
'AL-index, nT',
'AU-index, nT',
'SYM/D index, nT',
'SYM/H index, nT',
'ASY/D index, nT',
'ASY/H index, nT',
'PC(N) index',
'Magnetosonic mach number')
# Set the url
omni_url = 'https://spdf.gsfc.nasa.gov/pub/data/omni/'
if kwargs.get('high_res', True):
omni_url += 'high_res_omni/monthly_1min/'
col_names = OMNI_COLS # column names from spdf
# Initialize return dict
return_data = {}
return_data['times'] = []
......@@ -138,12 +99,11 @@ def _check_bad_omni_num(value_string):
in omni.
"""
for char in value_string:
if char != '9' and char != '.':
if char not in ('9', '.'):
return False
return True
def download_magnetogram_hmi(mag_time, hmi_map='hmi.B_720s', **kwargs):
"""Downloads HMI vector magnetogram fits files.
......@@ -209,7 +169,7 @@ def download_magnetogram_hmi(mag_time, hmi_map='hmi.B_720s', **kwargs):
}
client = drms.Client()
urls = get_urls[hmi_map](client, mag_time)
urls = get_urls[hmi_map](client, mag_time)
# Download data
if kwargs.get('verbose', False):
......@@ -250,7 +210,6 @@ def _get_urls_hmi_b_synoptic_small(client, mag_time):
generator that yields (datetime.datetime, str): Time of magnetogram,
suffix url of magnetogram
"""
import drms
try:
from sunpy.coordinates.sun import carrington_rotation_number
except ImportError as error:
......@@ -284,7 +243,7 @@ def _get_urls_hmi_b720(client, mag_time):
query_string += f'{str(mag_time.month).zfill(2)}.'
query_string += f'{str(mag_time.day).zfill(2)}_'
query_string += f'{str(mag_time.hour).zfill(2)}'
query_string += f'/1h]'
query_string += '/1h]'
data = client.query(query_string, key='T_REC', seg='field')
times = drms.to_datetime(data[0].T_REC)
nearest_time = _nearest(mag_time, times)
......@@ -293,7 +252,7 @@ def _get_urls_hmi_b720(client, mag_time):
in zip(times, data[1].field) if data_time == nearest_time)
return urls
def download_magnetogram_adapt(time, map_type='fixed', **kwargs):
"""This routine downloads GONG ADAPT magnetograms.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment