Note: The default ITS GitLab runner is a shared resource and is subject to slowdowns during heavy usage.
You can run your own GitLab runner that is dedicated just to your group if you need to avoid processing delays.

Commit 63f6dab4 authored by Qusai Al Shidi's avatar Qusai Al Shidi 💬
Browse files

lint fixes and found documentation method to show # symbol. Use `share/make_docs.sh`

parent 8038562c
# Table of Contents
* [swmfpy](#.swmfpy)
* [swmfpy.web](#.swmfpy.web)
* [get\_omni\_data](#.swmfpy.web.get_omni_data)
* [download\_magnetogram\_hmi](#.swmfpy.web.download_magnetogram_hmi)
* [download\_magnetogram\_adapt](#.swmfpy.web.download_magnetogram_adapt)
* [swmfpy.io](#.swmfpy.io)
* [read\_wdc\_ae](#.swmfpy.io.read_wdc_ae)
* [read\_wdc\_asy\_sym](#.swmfpy.io.read_wdc_asy_sym)
......@@ -9,10 +13,6 @@
* [swmfpy.paramin](#.swmfpy.paramin)
* [replace\_command](#.swmfpy.paramin.replace_command)
* [read\_command](#.swmfpy.paramin.read_command)
* [swmfpy.web](#.swmfpy.web)
* [get\_omni\_data](#.swmfpy.web.get_omni_data)
* [download\_magnetogram\_hmi](#.swmfpy.web.download_magnetogram_hmi)
* [download\_magnetogram\_adapt](#.swmfpy.web.download_magnetogram_adapt)
<a name=".swmfpy"></a>
## swmfpy
......@@ -36,6 +36,170 @@ These are not automatically imported. Might have extra dependancies.
*None yet.*
<a name=".swmfpy.web"></a>
## swmfpy.web
Tools to retrieve and send data on the web.
Here are a collection of tools to work with data on the internet. Thus,
this module mostly requires an internet connection.
<a name=".swmfpy.web.get_omni_data"></a>
#### get\_omni\_data
```python
get_omni_data(time_from, time_to, **kwargs)
```
Retrieve omni solar wind data over http.
This will download omni data from https://spdf.gsfc.nasa.gov/pub/data/omni
and put it into a dictionary. If your data is large, then make a csv and
use swmfpy.io.read_omni_data().
**Arguments**:
- `time_from` _datetime.datetime_ - The start time of the solar wind
data that you want to receive.
- `time_to` _datetime.datetime_ - The end time of the solar wind data
you want to receive.
**Returns**:
- `dict` - This will be a list of *all* columns
available in the omni data set.
**Examples**:
```python
import datetime
import swmfpy.web
storm_start = datetime.datetime(year=2000, month=1, day=1)
storm_end = datetime.datetime(year=2000, month=2, day=15)
data = swmfpy.web.get_omni_data(time_from=storm_start,
time_to=storm_end)
```
<a name=".swmfpy.web.download_magnetogram_hmi"></a>
#### download\_magnetogram\_hmi
```python
download_magnetogram_hmi(mag_time, hmi_map='hmi.B_720s', **kwargs)
```
Downloads HMI vector magnetogram fits files.
This will download vector magnetogram FITS files from
Joint Science Operations Center (JSOC) near a certain hour.
This unfortunately depends on sunpy and drms, if you don't have it try,
```bash
pip install -U --user sunpy drms
```
**Arguments**:
- `mag_time` _datetime.datetime_ - Time after which to find
vector magnetograms.
- `hmi_map` _str_ - JSOC prefix for hmi maps. Currently can only do
'hmi.B_720s' and 'hmi.b_synoptic.small'.
**kwargs:
- `download_dir` _str_ - Relative directory to download to.
- `verbose` _bool_ - (default False) print out the files it's downloading.
**Returns**:
- `str` - list of filenames downloaded.
**Raises**:
- `ImportError` - If module `drms` is not found.
- `FileNotFoundError` - If the JSOC doesn't have the magnetograms for that
time.
**Examples**:
```python
from swmfpy.web import download_magnetogram_hmi
import datetime as dt
# I am interested in the hmi vector magnetogram from 2014, 2, 18
time_mag = dt.datetime(2014, 2, 18, 10) # Around hour 10
# Calling it will download
filenames = download_magnetogram_hmi(mag_time=time_mag,
hmi_map='B_720s',
download_dir='mydir/')
# To see my list
print('The magnetograms I downloaded are:', filenames)
# You may call and ignore the file list
download_magnetogram_hmi(mag_time=time_mag,
hmi_map='b_synoptic_small',
download_dir='mydir')
```
<a name=".swmfpy.web.download_magnetogram_adapt"></a>
#### download\_magnetogram\_adapt
```python
download_magnetogram_adapt(time, map_type='fixed', **kwargs)
```
This routine downloads GONG ADAPT magnetograms.
Downloads ADAPT magnetograms from ftp://gong2.nso.edu/adapt/maps/gong/
to a local directory. It will download all maps with the regex file
pattern: adapt4[0,1]3*yyyymmddhh
**Arguments**:
- `time` _datetime.datetime_ - Time in which you want the magnetogram.
- `map_type` _str_ - (default: 'fixed')
Choose either 'fixed' or 'central' for
the map type you want.
**kwargs:
- `download_dir` _str_ - (default is current dir) Relative directory
where you want the maps to be downloaded.
**Returns**:
- `str` - First unzipped filename found.
**Raises**:
- `NotADirectoryError` - If the adapt maps directory
is not found on the server.
- `ValueError` - If map_type is not recognized.
(i.e. not 'fixed' or 'central')
- `FileNotFoundError` - If maps were not found.
**Examples**:
```python
import datetime as dt
# Use datetime objects for the time
time_flare = dt.datetime(2018, 2, 12, hour=10)
swmfpy.web.download_magnetogram_adapt(time=time_flare,
map_type='central',
download_dir='./mymaps/')
```
<a name=".swmfpy.io"></a>
## swmfpy.io
......@@ -186,7 +350,7 @@ Note, if you have repeat commands this will replace all the repeats.
**Examples**:
```python
change['`SOLARWINDFILE`'] = [['T', 'UseSolarWindFile'],
change['\#SOLARWINDFILE'] = [['T', 'UseSolarWindFile'],
['new_imf.dat', 'NameSolarWindFile']]
# This will overwrite PARAM.in
swmfpy.paramin.replace('PARAM.in.template', change)
......@@ -228,8 +392,8 @@ values for the parameters.
**Examples**:
```python
start_time = swmfpy.paramin.read_command('`STARTTIME`')
end_time = swmfpy.paramin.read_command('`ENDTIME`')
start_time = swmfpy.paramin.read_command('\#STARTTIME')
end_time = swmfpy.paramin.read_command('\#ENDTIME')
print('Starting month is ', start_time[1])
print('Ending month is ', end_time[1])
```
......@@ -238,167 +402,3 @@ values for the parameters.
this, try using the `num_of_values` keyword. This is helpful if your
PARAM.in is comment heavy.
<a name=".swmfpy.web"></a>
## swmfpy.web
Tools to retrieve and send data on the web.
Here are a collection of tools to work with data on the internet. Thus,
this module mostly requires an internet connection.
<a name=".swmfpy.web.get_omni_data"></a>
#### get\_omni\_data
```python
get_omni_data(time_from, time_to, **kwargs)
```
Retrieve omni solar wind data over http.
This will download omni data from https://spdf.gsfc.nasa.gov/pub/data/omni
and put it into a dictionary. If your data is large, then make a csv and
use swmfpy.io.read_omni_data().
**Arguments**:
- `time_from` _datetime.datetime_ - The start time of the solar wind
data that you want to receive.
- `time_to` _datetime.datetime_ - The end time of the solar wind data
you want to receive.
**Returns**:
- `dict` - This will be a list of *all* columns
available in the omni data set.
**Examples**:
```python
import datetime
import swmfpy.web
storm_start = datetime.datetime(year=2000, month=1, day=1)
storm_end = datetime.datetime(year=2000, month=2, day=15)
data = swmfpy.web.get_omni_data(time_from=storm_start,
time_to=storm_end)
```
<a name=".swmfpy.web.download_magnetogram_hmi"></a>
#### download\_magnetogram\_hmi
```python
download_magnetogram_hmi(mag_time, hmi_map, **kwargs)
```
Downloads HMI vector magnetogram fits files.
This will download vector magnetogram FITS files from
Joint Science Operations Center (JSOC) near a certain hour.
This unfortunately depends on sunpy and drms, if you don't have it try,
```bash
pip install -U --user sunpy drms
```
**Arguments**:
- `mag_time` _datetime.datetime_ - Time after which to find
vector magnetograms.
- `hmi_map` _str_ - JSOC prefix for hmi maps. Currently can only do
'hmi.B_720s' and 'hmi.b_synoptic.small'.
**kwargs:
- `download_dir` _str_ - Relative directory to download to.
- `verbose` _bool_ - (default False) print out the files it's downloading.
**Returns**:
- `str` - list of filenames downloaded.
**Raises**:
- `ImportError` - If module `drms` is not found.
- `FileNotFoundError` - If the JSOC doesn't have the magnetograms for that
time.
**Examples**:
```python
from swmfpy.web import download_magnetogram_hmi
import datetime as dt
# I am interested in the hmi vector magnetogram from 2014, 2, 18
time_mag = dt.datetime(2014, 2, 18, 10) # Around hour 10
# Calling it will download
filenames = download_magnetogram_hmi(mag_time=time_mag,
hmi_map='B_720s',
download_dir='mydir/')
# To see my list
print('The magnetograms I downloaded are:', filenames)
# You may call and ignore the file list
download_magnetogram_hmi(mag_time=time_mag,
hmi_map='b_synoptic_small',
download_dir='mydir')
```
<a name=".swmfpy.web.download_magnetogram_adapt"></a>
#### download\_magnetogram\_adapt
```python
download_magnetogram_adapt(time, map_type='fixed', **kwargs)
```
This routine downloads GONG ADAPT magnetograms.
Downloads ADAPT magnetograms from ftp://gong2.nso.edu/adapt/maps/gong/
to a local directory. It will download all maps with the regex file
pattern: adapt4[0,1]3*yyyymmddhh
**Arguments**:
- `time` _datetime.datetime_ - Time in which you want the magnetogram.
- `map_type` _str_ - (default: 'fixed')
Choose either 'fixed' or 'central' for
the map type you want.
**kwargs:
- `download_dir` _str_ - (default is current dir) Relative directory
where you want the maps to be downloaded.
**Returns**:
- `str` - First unzipped filename found.
**Raises**:
- `NotADirectoryError` - If the adapt maps directory
is not found on the server.
- `ValueError` - If map_type is not recognized.
(i.e. not 'fixed' or 'central')
- `FileNotFoundError` - If maps were not found.
**Examples**:
```python
import datetime as dt
# Use datetime objects for the time
time_flare = dt.datetime(2018, 2, 12, hour=10)
swmfpy.web.download_magnetogram_adapt(time=time_flare,
map_type='central',
download_dir='./mymaps/')
```
......@@ -35,7 +35,9 @@ import swmfpy
Documentation
-------------
Documentation is included as docstrings. Use python's *dir()* and *help()* inbuilt functions to see documentation.
An auto-documented version can be found [here](DOCUMENTATION.markdown).
However, documentation is included as docstrings. Use python's *dir()* and *help()* inbuilt functions to see documentation.
```python
import swmfpy
......@@ -43,10 +45,6 @@ help(swmfpy) # To see list of functions
help(swmfpy.io.read_gm_log) # To see the function documentation
```
However if you would like an auto-documented version (uses exact same text as the help() function output), go [here](DOCUMENTATION.markdown).
*Note*: The autodoc tool has troubles escaping the `#` symbol so examples with `#COMMAND`s in `PARAM.in` files will not show the symbol. Perhaps I can find a fix but the `help()` pages are always better and up to date.
Issues
------
......
#!/usr/bin/env bash
# Script to make the docs for swmfpy.
# Requires pydoc-markdown
# run `pip install pydoc-markdown --user`
# then run this script in the project root directory
pydoc-markdown -v -I. --render-toc -m swmfpy -m swmfpy.web -m swmfpy.io -m swmfpy.paramin -m swmfpy.tools | sed 's/\\043/\\#/g' - > DOCUMENTATION.markdown
"""A collection of tools to read, write, visualize with the
Space Weather Modeling Framework (SWMF).
Modules
-------
### Modules
These are automatically imported.
......@@ -10,8 +9,7 @@ These are automatically imported.
- `swmfpy.paramin` PARAM.in editing tools.
- `swmfpy.web` Internet data downloading/uploading tools.
Extra Modules
-------------
### Extra Modules
These are not automatically imported. Might have extra dependancies.
......@@ -29,3 +27,47 @@ assert sys.version_info >= (3, 6), "swmfpy requires Python >=3.6. Sorry :(."
from . import paramin
from . import io
from . import web
# This is straight from the format guide on spdf
OMNI_COLS = ('ID for IMF spacecraft',
'ID for SW Plasma spacecraft',
'# of points in IMF averages',
'# of points in Plasma averages',
'Percent interp',
'Timeshift, sec',
'RMS, Timeshift',
'RMS, Phase front normal',
'Time btwn observations, sec',
'Field magnitude average, nT',
'Bx, nT (GSE, GSM)',
'By, nT (GSE)',
'Bz, nT (GSE)',
'By, nT (GSM)',
'Bz, nT (GSM)',
'RMS SD B scalar, nT',
'RMS SD field vector, nT',
'Flow speed, km/s',
'Vx Velocity, km/s, GSE',
'Vy Velocity, km/s, GSE',
'Vz Velocity, km/s, GSE',
'Proton Density, n/cc',
'Temperature, K',
'Flow pressure, nPa',
'Electric field, mV/m',
'Plasma beta',
'Alfven mach number',
'X(s/c), GSE, Re',
'Y(s/c), GSE, Re',
'Z(s/c), GSE, Re',
'BSN location, Xgse, Re',
'BSN location, Ygse, Re',
'BSN location, Zgse, Re',
'AE-index, nT',
'AL-index, nT',
'AU-index, nT',
'SYM/D index, nT',
'SYM/H index, nT',
'ASY/D index, nT',
'ASY/H index, nT',
'PC(N) index',
'Magnetosonic mach number')
"""Input/Output tools
"""### Input/Output tools
Here are tools to read and write files relating to SWMF.
TODO: Move pandas dependancy elsewhere.
"""
import datetime as dt
......
""" These tools are to help script the editing of PARAM.in files.
"""### Editing PARAM.in files
These tools are to help script the editing and reading of PARAM.in files.
"""
__all__ = [
'read_command',
......@@ -30,7 +32,7 @@ def replace_command(parameters, input_file, output_file='PARAM.in'):
Examples:
```python
change['#SOLARWINDFILE'] = [['T', 'UseSolarWindFile'],
change['\043SOLARWINDFILE'] = [['T', 'UseSolarWindFile'],
['new_imf.dat', 'NameSolarWindFile']]
# This will overwrite PARAM.in
swmfpy.paramin.replace('PARAM.in.template', change)
......@@ -92,8 +94,8 @@ def read_command(command, paramin_file='PARAM.in', **kwargs):
Examples:
```python
start_time = swmfpy.paramin.read_command('#STARTTIME')
end_time = swmfpy.paramin.read_command('#ENDTIME')
start_time = swmfpy.paramin.read_command('\043STARTTIME')
end_time = swmfpy.paramin.read_command('\043ENDTIME')
print('Starting month is ', start_time[1])
print('Ending month is ', end_time[1])
```
......@@ -154,7 +156,7 @@ def _get_command(line):
line (str, list, tuple): The line in the PARAM.in file.
Returns:
(str): '#COMMAND' if found and None if not.
(str): '\043COMMAND' if found and None if not.
"""
if isinstance(line, (str, list, tuple)): # Raises type error otherwise
if isinstance(line, str):
......
"""Tools to retrieve and send data on the web.
"""### Tools to download/upload data on the web
Here are a collection of tools to work with data on the internet. Thus,
this module mostly requires an internet connection.
this module mostly requires an internet connection. Which on some
supercomputers would be turned off during a job run. In scripts, make sure to
use these to preprocess before submitting jobs.
"""
__author__ = 'Qusai Al Shidi'
__email__ = 'qusai@umich.edu'
import datetime as dt
import urllib
from .__init__ import OMNI_COLS
from .tools import _import_error_string, _nearest
......@@ -44,55 +47,13 @@ def get_omni_data(time_from, time_to, **kwargs):
from dateutil import rrule
# This is straight from the format guide on spdf
col_names = ('ID for IMF spacecraft',
'ID for SW Plasma spacecraft',
'# of points in IMF averages',
'# of points in Plasma averages',
'Percent interp',
'Timeshift, sec',
'RMS, Timeshift',
'RMS, Phase front normal',
'Time btwn observations, sec',
'Field magnitude average, nT',
'Bx, nT (GSE, GSM)',
'By, nT (GSE)',
'Bz, nT (GSE)',
'By, nT (GSM)',
'Bz, nT (GSM)',
'RMS SD B scalar, nT',
'RMS SD field vector, nT',
'Flow speed, km/s',
'Vx Velocity, km/s, GSE',
'Vy Velocity, km/s, GSE',
'Vz Velocity, km/s, GSE',
'Proton Density, n/cc',
'Temperature, K',
'Flow pressure, nPa',
'Electric field, mV/m',
'Plasma beta',
'Alfven mach number',
'X(s/c), GSE, Re',
'Y(s/c), GSE, Re',
'Z(s/c), GSE, Re',
'BSN location, Xgse, Re',
'BSN location, Ygse, Re',
'BSN location, Zgse, Re',
'AE-index, nT',
'AL-index, nT',
'AU-index, nT',
'SYM/D index, nT',
'SYM/H index, nT',
'ASY/D index, nT',
'ASY/H index, nT',
'PC(N) index',
'Magnetosonic mach number')
# Set the url
omni_url = 'https://spdf.gsfc.nasa.gov/pub/data/omni/'
if kwargs.get('high_res', True):
omni_url += 'high_res_omni/monthly_1min/'
col_names = OMNI_COLS # column names from spdf
# Initialize return dict
return_data = {}
return_data['times'] = []
......@@ -138,12 +99,11 @@ def _check_bad_omni_num(value_string):
in omni.
"""
for char in value_string:
if char != '9' and char != '.':
if char not in ('9', '.'):
return False
return True
def download_magnetogram_hmi(mag_time, hmi_map='hmi.B_720s', **kwargs):
"""Downloads HMI vector magnetogram fits files.
......@@ -209,7 +169,7 @@ def download_magnetogram_hmi(mag_time, hmi_map='hmi.B_720s', **kwargs):
}
client = drms.Client()
urls = get_urls[hmi_map](client, mag_time)
urls = get_urls[hmi_map](client, mag_time)
# Download data
if kwargs.get('verbose', False):
......@@ -250,7 +210,6 @@ def _get_urls_hmi_b_synoptic_small(client, mag_time):
generator that yields (datetime.datetime, str): Time of magnetogram,
suffix url of magnetogram
"""
import drms
try:
from sunpy.coordinates.sun import carrington_rotation_number
except ImportError as error:
......@@ -284,7 +243,7 @@ def _get_urls_hmi_b720(client, mag_time):
query_string += f'{str(mag_time.month).zfill(2)}.'
query_string += f'{str(mag_time.day).zfill(2)}_'
query_string += f'{str(mag_time.hour).zfill(2)}'
query_string += f'/1h]'
query_string += '/1h]'
data = client.query(query_string, key='T_REC', seg='field')
times = drms.to_datetime(data[0].T_REC)
nearest_time = _nearest(mag_time, times)
......@@ -293,7 +252,7 @@ def _get_urls_hmi_b720(client, mag_time):
in zip(times, data[1].field) if data_time == nearest_time<