Note: The default ITS GitLab runner is a shared resource and is subject to slowdowns during heavy usage.
You can run your own GitLab runner that is dedicated just to your group if you need to avoid processing delays.

Commit 48200d70 authored by Qusai Al Shidi's avatar Qusai Al Shidi 💬
Browse files

Merge branch 'imf-f107' into 'master'

Imf f107

See merge request !2
parents d4a5c35f 3c757873
# Table of Contents
* [swmfpy](#.swmfpy)
* [swmfpy.tools](#.swmfpy.tools)
* [swmfpy.io](#.swmfpy.io)
* [read\_wdc\_ae](#.swmfpy.io.read_wdc_ae)
* [read\_wdc\_asy\_sym](#.swmfpy.io.read_wdc_asy_sym)
* [read\_gm\_log](#.swmfpy.io.read_gm_log)
* [write\_imf\_from\_omni](#.swmfpy.write_imf_from_omni)
* [swmfpy.web](#.swmfpy.web)
* [get\_omni\_data](#.swmfpy.web.get_omni_data)
* [download\_magnetogram\_hmi](#.swmfpy.web.download_magnetogram_hmi)
* [download\_magnetogram\_adapt](#.swmfpy.web.download_magnetogram_adapt)
* [swmfpy.io](#.swmfpy.io)
* [write\_imf\_input](#.swmfpy.io.write_imf_input)
* [read\_wdc\_ae](#.swmfpy.io.read_wdc_ae)
* [read\_wdc\_asy\_sym](#.swmfpy.io.read_wdc_asy_sym)
* [read\_gm\_log](#.swmfpy.io.read_gm_log)
* [swmfpy.paramin](#.swmfpy.paramin)
* [replace\_command](#.swmfpy.paramin.replace_command)
* [read\_command](#.swmfpy.paramin.read_command)
* [swmfpy.tools](#.swmfpy.tools)
<a name=".swmfpy"></a>
## swmfpy
......@@ -34,114 +36,36 @@ These are not automatically imported. Might have extra dependancies.
*None yet.*
<a name=".swmfpy.tools"></a>
## swmfpy.tools
Tools to be used in swmfpy functions and classes. Some of the functions are
*hidden functions*.
<a name=".swmfpy.io"></a>
## swmfpy.io
### Input/Output tools
Here are tools to read and write files relating to SWMF.
<a name=".swmfpy.io.read_wdc_ae"></a>
#### read\_wdc\_ae
<a name=".swmfpy.write_imf_from_omni"></a>
#### write\_imf\_from\_omni
```python
read_wdc_ae(wdc_filename)
write_imf_from_omni(time_from, time_to, filename='IMF.dat', **kwargs)
```
Read an auroral electrojet (AE) indeces from Kyoto's World Data Center
text file into a dictionary of lists.
Writes an IMF.dat file for the geospace model runs for a specific time
period.
**Arguments**:
- `wdc_filename` _str_ - Filename of wdc data from
http://wdc.kugi.kyoto-u.ac.jp/
**Returns**:
- `dict` - Auroral indeces 'AL', 'AE', 'AO', 'AU'
- `datetime.datetime` - 'times'
- `int` - 'values'
<a name=".swmfpy.io.read_wdc_asy_sym"></a>
#### read\_wdc\_asy\_sym
```python
read_wdc_asy_sym(wdc_filename)
```
Reads a WDC file for ASY/SYM data.
Reads an ASY/SYM file downloaded from
http://wdc.kugi.kyoto-u.ac.jp/aeasy/index.html
and puts it into a dictionary.
**Arguments**:
- `wdc_filename` _str_ - Relative filename (or file handle no.) to read.
**Returns**:
- `dict` - of values. {'[ASY-SYM]-[D-H]': 'times': [], 'values': []}
**Examples**:
```python
indeces = swmfpy.io.read_wdc_asy_sym('wdc.dat')
# Plot data
plt.plot(indeces['SYM-H']['times'],
indeces['SYM-H']['values'],
label='SYM-H [nT]'
)
plt.xlabel('Time [UT]')
```
Important to note if there is bad data it will be filled as None.
<a name=".swmfpy.io.read_gm_log"></a>
#### read\_gm\_log
```python
read_gm_log(filename, colnames=None, dtypes=None, index_time=True)
```
Make a dictionary out of the indeces outputted
from the GM model log.
**Arguments**:
- `filename` _str_ - The relative filename as a string. (or file handle no.)
- `colnames` _[str]_ - (default: None) Supply the name of the columns.
If None it will use second line
of log file.
- `dtypes` _[types]_ - (default: None) Provide types for the columns, if
None then all will be float.
- `index_time` _bool_ - (default: True) Make a column of dt.datetime objects
in dictionary key 'Time [UT]'.
**Returns**:
- `dict` - Dictionary of the log file
- `time_from` _datetime.datetime_ - Time to begin omni data retrieval
- `time_to` _datetime.datetime_ - Time to end omni data retrieval
- `filename` _str_ - The filename for the dat file, defaults to 'IMF.dat'.
**kwargs:
see `swmfpy.io.write_imf_input()` and `swmfpy.web.get_omni_data()`
**Examples**:
To plot AL and Dst get the log files
Using this function is simple:
```python
geo = swmfpy.io.read_gm_log('run/GM/IO2/geoindex_e20140215-100500.log')
dst = swmfpy.io.read_gm_log('run/GM/IO2/log_e20140215-100500.log')
# Plot AL indeces
plt.plot(geo['times', geo['AL'])
import swmfpy
import datetime as dt
times = (dt.datetime(2014, 2, 2), dt.datetime(2014, 2, 4))
# Usually the kwargs are unecessary
swmfpy.write_imf_input(*times)
# Sometimes this
swmfpy.write_imf_input(*times, filename='run/IMF.dat')
```
<a name=".swmfpy.web"></a>
......@@ -173,6 +97,14 @@ use swmfpy.io.read_omni_data().
data that you want to receive.
- `time_to` _datetime.datetime_ - The end time of the solar wind data
you want to receive.
**kwargs:
- `original_colnames` _bool_ - Use the original column names from the
spdf specification. The alternative is
nicer and shorter names. Defaults to
False.
- `resolution` _str_ - (default: 'high') Here you can choose 'high' or
'low' resolution omni data. Some columns appear
in one but not the other.
**Returns**:
......@@ -191,6 +123,10 @@ use swmfpy.io.read_omni_data().
storm_end = datetime.datetime(year=2000, month=2, day=15)
data = swmfpy.web.get_omni_data(time_from=storm_start,
time_to=storm_end)
# or for low res
data = swmfpy.web.get_omni_data(time_from=storm_start,
time_to=storm_end,
resolution='low')
```
<a name=".swmfpy.web.download_magnetogram_hmi"></a>
......@@ -310,6 +246,151 @@ pattern: adapt4[0,1]3*yyyymmddhh
download_dir='./mymaps/')
```
<a name=".swmfpy.io"></a>
## swmfpy.io
### Input/Output tools
Here are tools to read and write files relating to SWMF.
<a name=".swmfpy.io.write_imf_input"></a>
#### write\_imf\_input
```python
write_imf_input(imf_data, filename='IMF.dat', **kwargs)
```
Creates the IMF.dat input file for the SWMF BATS-R-US geospace model.
`imf_data` must be a dictionary of array_like objects with same length
in data. In swmfpy Pythonic versions are always preferred so the 'times'
must be `datetime.datetime` array.
imf_data = dict(times, bx, by, bz, vx, vy, vz, density, temperature)
**Arguments**:
- `imf_data` _dict_ - This dictionary contains the solar wind data.
- `filename` _str_ - (default: 'IMF.dat') Filename to write to.
**kwargs:
- `commands` _[str]_ - (default: None) List of commands to write to imf
input file (indexed by line then by tabs on same
line). *Note*: Must be a list if have one command
str.
**Raises**:
- `TypeError` - If commands is not a list or tuple. It must be at least a
one element list of strings.
- `AssertionError` - If inputs aren't prepared properly (key names)
**Examples**:
```python
from swmfpy.io import write_imf_input
# Prepare imf dictionary: imf_data
write_imf_input(imf_data, filename='run/IMF.dat')
```
<a name=".swmfpy.io.read_wdc_ae"></a>
#### read\_wdc\_ae
```python
read_wdc_ae(wdc_filename)
```
Read an auroral electrojet (AE) indeces from Kyoto's World Data Center
text file into a dictionary of lists.
**Arguments**:
- `wdc_filename` _str_ - Filename of wdc data from
http://wdc.kugi.kyoto-u.ac.jp/
**Returns**:
- `dict` - Auroral indeces 'AL', 'AE', 'AO', 'AU'
- `datetime.datetime` - 'times'
- `int` - 'values'
<a name=".swmfpy.io.read_wdc_asy_sym"></a>
#### read\_wdc\_asy\_sym
```python
read_wdc_asy_sym(wdc_filename)
```
Reads a WDC file for ASY/SYM data.
Reads an ASY/SYM file downloaded from
http://wdc.kugi.kyoto-u.ac.jp/aeasy/index.html
and puts it into a dictionary.
**Arguments**:
- `wdc_filename` _str_ - Relative filename (or file handle no.) to read.
**Returns**:
- `dict` - of values. {'[ASY-SYM]-[D-H]': 'times': [], 'values': []}
**Examples**:
```python
indeces = swmfpy.io.read_wdc_asy_sym('wdc.dat')
# Plot data
plt.plot(indeces['SYM-H']['times'],
indeces['SYM-H']['values'],
label='SYM-H [nT]'
)
plt.xlabel('Time [UT]')
```
Important to note if there is bad data it will be filled as None.
<a name=".swmfpy.io.read_gm_log"></a>
#### read\_gm\_log
```python
read_gm_log(filename, colnames=None, dtypes=None, index_time=True)
```
Make a dictionary out of the indeces outputted
from the GM model log.
**Arguments**:
- `filename` _str_ - The relative filename as a string. (or file handle no.)
- `colnames` _[str]_ - (default: None) Supply the name of the columns.
If None it will use second line
of log file.
- `dtypes` _[types]_ - (default: None) Provide types for the columns, if
None then all will be float.
- `index_time` _bool_ - (default: True) Make a column of dt.datetime objects
in dictionary key 'Time [UT]'.
**Returns**:
- `dict` - Dictionary of the log file
**Examples**:
To plot AL and Dst get the log files
```python
geo = swmfpy.io.read_gm_log('run/GM/IO2/geoindex_e20140215-100500.log')
dst = swmfpy.io.read_gm_log('run/GM/IO2/log_e20140215-100500.log')
# Plot AL indeces
plt.plot(geo['times', geo['AL'])
```
<a name=".swmfpy.paramin"></a>
## swmfpy.paramin
......@@ -331,7 +412,7 @@ Note, if you have repeat commands this will replace all the repeats.
**Arguments**:
- `parameters` _dict_ - Dictionary of strs with format
replace = '#COMMAND': ['value', 'comments', ...]
replace = '\#COMMAND': ['value', 'comments', ...]
This is case sensitive.
- `input_file` _str_ - String of PARAM.in file name.
- `output_file` _str_ - (default 'PARAM.in') The output file to write to.
......@@ -350,7 +431,7 @@ Note, if you have repeat commands this will replace all the repeats.
**Examples**:
```python
change['#SOLARWINDFILE'] = [['T', 'UseSolarWindFile'],
change['\#SOLARWINDFILE'] = [['T', 'UseSolarWindFile'],
['new_imf.dat', 'NameSolarWindFile']]
# This will overwrite PARAM.in
swmfpy.paramin.replace('PARAM.in.template', change)
......@@ -365,12 +446,12 @@ read_command(command, paramin_file='PARAM.in', **kwargs)
Get parameters of a certain command in PARAM.in file.
This will find the #COMMAND and return a list of
This will find the \#COMMAND and return a list of
values for the parameters.
**Arguments**:
- `command` _str_ - This is the #COMMAND you're looking for.
- `command` _str_ - This is the \#COMMAND you're looking for.
- `paramin_file` _str_ - (default: 'PARAM.in') The file in which you're
looking for the command values.
**kwargs:
......@@ -380,20 +461,20 @@ values for the parameters.
**Returns**:
- `list` - Values found for the #COMMAND in file. Index 0 is
#COMMAND and the values follow (1 for first argument...)
- `list` - Values found for the \#COMMAND in file. Index 0 is
\#COMMAND and the values follow (1 for first argument...)
**Raises**:
- `ValueError` - When the #COMMAND is not found.
- `ValueError` - When the \#COMMAND is not found.
**Examples**:
```python
start_time = swmfpy.paramin.read_command('#STARTTIME')
end_time = swmfpy.paramin.read_command('#ENDTIME')
start_time = swmfpy.paramin.read_command('\#STARTTIME')
end_time = swmfpy.paramin.read_command('\#ENDTIME')
print('Starting month is ', start_time[1])
print('Ending month is ', end_time[1])
```
......@@ -402,3 +483,9 @@ values for the parameters.
this, try using the `num_of_values` keyword. This is helpful if your
PARAM.in is comment heavy.
<a name=".swmfpy.tools"></a>
## swmfpy.tools
Tools to be used in swmfpy functions and classes. Some of the functions are
*hidden functions*.
......@@ -15,17 +15,19 @@ Clone into the directory you want to use it.
*Note*: swmfpy also is part of the SWMF and gets cloned into `SWMF/share/Python`. However, if you would like to [develop](CONTRIBUTING.markdown) for swmfpy or have a local copy do the following:
```bash
# Skip this if using it in SWMF directory.
git clone https://gitlab.umich.edu/swmf_sofware/swmfpy.git /path/to/my/dir
```
Then go to its directory and run `setup.py` make sure to include `--user`.
Then go to its directory and run [pip](https://pip.pypa.io/en/stable/) to install. Make sure to include `--user`.
```bash
cd /path/to/swmfpy
python3 -m pip install setuptools wheel twine --user # If you don't have these
python3 setup.py install --user
cd SWMF/share/Python # or your clone directory
pip install --user .
```
*Note*: Depending on your system [pip](https://pip.pypa.io/en/stable/) may be ran in several ways: `pip`, `pip3`, or `python3 -m pip`
Then import it into your python project.
```python
......
......@@ -5,8 +5,11 @@ swmfpy.
import setuptools
with open('README.markdown') as readme_fh:
LONG_DESCRIPTION = readme_fh.read()
with open('README.markdown') as fh_readme:
LONG_DESCRIPTION = fh_readme.read()
with open('requirements.txt') as fh_requirements:
REQUIREMENTS = list(fh_requirements)
setuptools.setup(
name='swmfpy',
......@@ -24,5 +27,6 @@ setuptools.setup(
'License :: OSI Approved :: MIT License',
'Operating System :: OS Independent',
],
install_requires=REQUIREMENTS,
python_requires='>=3.6'
)
......@@ -4,5 +4,5 @@
# run `pip install pydoc-markdown --user`
# then run this script in the project root directory
pydoc-markdown -v -I. --render-toc -m swmfpy -m swmfpy.web -m swmfpy.io -m swmfpy.paramin -m swmfpy.tools | sed 's/\\043/#/g' - > DOCUMENTATION.markdown
pydoc-markdown --py3 -v -I. --render-toc -m swmfpy -m swmfpy.web -m swmfpy.io -m swmfpy.paramin -m swmfpy.tools | sed 's/\\043/#/g' - > DOCUMENTATION.markdown
......@@ -23,51 +23,42 @@ __email__ = 'qusai@umich.edu'
import sys
assert sys.version_info >= (3, 6), "swmfpy requires Python >=3.6. Sorry :(."
from . import paramin
from . import io
from . import web
assert sys.version_info >= (3, 6), "swmfpy requires Python >=3.6. Sorry :(."
def write_imf_from_omni(time_from, time_to, filename='IMF.dat', **kwargs):
"""Writes an IMF.dat file for the geospace model runs for a specific time
period.
Args:
time_from (datetime.datetime): Time to begin omni data retrieval
time_to (datetime.datetime): Time to end omni data retrieval
filename (str): The filename for the dat file, defaults to 'IMF.dat'.
**kwargs:
see #swmfpy.io.write_imf_input() and #swmfpy.web.get_omni_data()
# This is straight from the format guide on spdf
OMNI_COLS = ('ID for IMF spacecraft',
'ID for SW Plasma spacecraft',
'# of points in IMF averages',
'# of points in Plasma averages',
'Percent interp',
'Timeshift, sec',
'RMS, Timeshift',
'RMS, Phase front normal',
'Time btwn observations, sec',
'Field magnitude average, nT',
'Bx, nT (GSE, GSM)',
'By, nT (GSE)',
'Bz, nT (GSE)',
'By, nT (GSM)',
'Bz, nT (GSM)',
'RMS SD B scalar, nT',
'RMS SD field vector, nT',
'Flow speed, km/s',
'Vx Velocity, km/s, GSE',
'Vy Velocity, km/s, GSE',
'Vz Velocity, km/s, GSE',
'Proton Density, n/cc',
'Temperature, K',
'Flow pressure, nPa',
'Electric field, mV/m',
'Plasma beta',
'Alfven mach number',
'X(s/c), GSE, Re',
'Y(s/c), GSE, Re',
'Z(s/c), GSE, Re',
'BSN location, Xgse, Re',
'BSN location, Ygse, Re',
'BSN location, Zgse, Re',
'AE-index, nT',
'AL-index, nT',
'AU-index, nT',
'SYM/D index, nT',
'SYM/H index, nT',
'ASY/D index, nT',
'ASY/H index, nT',
'PC(N) index',
'Magnetosonic mach number')
Examples:
Using this function is simple:
```python
import swmfpy
import datetime as dt
times = (dt.datetime(2014, 2, 2), dt.datetime(2014, 2, 4))
# Usually the kwargs are unecessary
swmfpy.write_imf_input(*times)
# Sometimes this
swmfpy.write_imf_input(*times, filename='run/IMF.dat')
```
"""
omni_data = web.get_omni_data(time_from, time_to, **kwargs)
commands = ['#COOR', 'GSE']
if kwargs.get('commands', None):
kwargs['commands'] += commands
else:
kwargs['commands'] = commands
column_keys = ['times',
'bx', 'by_gse', 'bz_gse', 'vx_gse', 'vy_gse', 'vz_gse',
'density', 'temperature']
io.write_imf_input(omni_data, filename, column_keys=column_keys, **kwargs)
......@@ -4,6 +4,83 @@ Here are tools to read and write files relating to SWMF.
"""
import datetime as dt
from .tools import _make_line
def write_imf_input(imf_data, filename='IMF.dat', **kwargs):
"""Creates the IMF.dat input file for the SWMF BATS-R-US geospace model.
`imf_data` must be a dictionary of array_like objects with same length
in data. In swmfpy Pythonic versions are always preferred so the 'times'
must be `datetime.datetime` array.
imf_data = dict(times, bx, by, bz, vx, vy, vz, density, temperature)
Args:
imf_data (dict): This dictionary contains the solar wind data.
filename (str): (default: 'IMF.dat') Filename to write to.
**kwargs:
commands ([str]): (default: None) List of commands to write to imf
input file (indexed by line then by tabs on same
line). *Note*: Must be a list if have one command
str.
Raises:
TypeError: If commands is not a list or tuple. It must be at least a
one element list of strings.
AssertionError: If inputs aren't prepared properly (key names)
Examples:
```python
from swmfpy.io import write_imf_input
# Prepare imf dictionary: imf_data
write_imf_input(imf_data, filename='run/IMF.dat')
```
"""
columns_dat = ['year', 'month', 'day', 'hour', 'min', 'sec', 'msec',
'bx', 'by', 'bz', 'vx', 'vy', 'vz',
'density', 'temperature']
column_keys = ['times',
'bx', 'by', 'bz', 'vx', 'vy', 'vz',
'density', 'temperature']
if kwargs.get('column_keys', None):
column_keys = kwargs.get('column_keys', column_keys)
def _time_repr(time_raw):
'Represent time in a suitable format'
raw_str = dt.datetime.strftime(time_raw, '%Y %m %d %H %M %S %f')[:-3]
return raw_str.split()
with open(filename, 'w') as file_imf:
# header
file_imf.write('Made with swmfpy ')
file_imf.write('(https://gitlab.umich.edu/swmf_software/swmfpy)\n\n')
# write commands
commands = kwargs.get('commands', None)
if commands:
try:
isinstance(commands, (list, tuple))
except TypeError:
raise TypeError(f'{__name__}: commands must be list or tuple')
for command in commands:
file_imf.write(_make_line(command)+'\n')
# write dat file
def justified(str_list, width=7):
'Returns justified'
return [s.rjust(width) for s in str_list]
file_imf.write('\n'+' '.join(justified(columns_dat)))
file_imf.write('\n#START\n')
lines = []
for index, _time in enumerate(imf_data[column_keys[0]]):
line = _time_repr(_time)
for key in column_keys[1:]:
line += [str(round(imf_data[key][index], 2))]
line = ' '.join(justified(line))
lines += line + '\n'
file_imf.writelines(lines)
def read_wdc_ae(wdc_filename):
......