Note: The default ITS GitLab runner is a shared resource and is subject to slowdowns during heavy usage.
You can run your own GitLab runner that is dedicated just to your group if you need to avoid processing delays.

Commit 8acb8c3e authored by Qusai Al Shidi's avatar Qusai Al Shidi 💬
Browse files

Version 2020.1. Changed download_magnetogram_hmi to take a new argument for...

Version 2020.1. Changed download_magnetogram_hmi to take a new argument for either B_720s or b_synoptic_small which is needed.
parent 3a10227e
# Table of Contents # Table of Contents
* [swmfpy](#.swmfpy) * [swmfpy](#.swmfpy)
* [swmfpy.web](#.swmfpy.web)
* [get\_omni\_data](#.swmfpy.web.get_omni_data)
* [download\_magnetogram\_hmi](#.swmfpy.web.download_magnetogram_hmi)
* [download\_magnetogram\_adapt](#.swmfpy.web.download_magnetogram_adapt)
* [swmfpy.io](#.swmfpy.io) * [swmfpy.io](#.swmfpy.io)
* [read\_wdc\_ae](#.swmfpy.io.read_wdc_ae) * [read\_wdc\_ae](#.swmfpy.io.read_wdc_ae)
* [read\_wdc\_asy\_sym](#.swmfpy.io.read_wdc_asy_sym) * [read\_wdc\_asy\_sym](#.swmfpy.io.read_wdc_asy_sym)
* [read\_gm\_log](#.swmfpy.io.read_gm_log) * [read\_gm\_log](#.swmfpy.io.read_gm_log)
* [swmfpy.tools](#.swmfpy.tools)
* [swmfpy.paramin](#.swmfpy.paramin) * [swmfpy.paramin](#.swmfpy.paramin)
* [replace\_command](#.swmfpy.paramin.replace_command) * [replace\_command](#.swmfpy.paramin.replace_command)
* [read\_command](#.swmfpy.paramin.read_command) * [read\_command](#.swmfpy.paramin.read_command)
* [swmfpy.web](#.swmfpy.web)
* [get\_omni\_data](#.swmfpy.web.get_omni_data)
* [download\_magnetogram\_hmi](#.swmfpy.web.download_magnetogram_hmi)
* [download\_magnetogram\_adapt](#.swmfpy.web.download_magnetogram_adapt)
<a name=".swmfpy"></a> <a name=".swmfpy"></a>
## swmfpy ## swmfpy
...@@ -35,165 +36,6 @@ These are not automatically imported. Might have extra dependancies. ...@@ -35,165 +36,6 @@ These are not automatically imported. Might have extra dependancies.
*None yet.* *None yet.*
<a name=".swmfpy.web"></a>
## swmfpy.web
Tools to retrieve and send data on the web.
Here are a collection of tools to work with data on the internet. Thus,
this module mostly requires an internet connection.
<a name=".swmfpy.web.get_omni_data"></a>
#### get\_omni\_data
```python
get_omni_data(time_from, time_to, **kwargs)
```
Retrieve omni solar wind data over http.
This will download omni data from https://spdf.gsfc.nasa.gov/pub/data/omni
and put it into a dictionary. If your data is large, then make a csv and
use swmfpy.io.read_omni_data().
**Arguments**:
- `time_from` _datetime.datetime_ - The start time of the solar wind
data that you want to receive.
- `time_to` _datetime.datetime_ - The end time of the solar wind data
you want to receive.
**Returns**:
- `dict` - This will be a list of *all* columns
available in the omni data set.
**Examples**:
```python
import datetime
import swmfpy.web
storm_start = datetime.datetime(year=2000, month=1, day=1)
storm_end = datetime.datetime(year=2000, month=2, day=15)
data = swmfpy.web.get_omni_data(time_from=storm_start,
time_to=storm_end)
```
<a name=".swmfpy.web.download_magnetogram_hmi"></a>
#### download\_magnetogram\_hmi
```python
download_magnetogram_hmi(mag_time, **kwargs)
```
Downloads HMI vector magnetogram fits files.
This will download vector magnetogram FITS files from
Joint Science Operations Center (JSOC) near a certain hour.
This unfortunately depends on sunpy/drms, if you don't have it try,
```bash
pip install -U --user drms
```
**Arguments**:
- `mag_time` _datetime.datetime_ - Time after which to find
vector magnetograms.
**kwargs:
- `download_dir` _str_ - Relative directory to download to.
- `verbose` _bool_ - (default False) print out the files it's downloading.
**Returns**:
- `str` - list of filenames downloaded.
**Raises**:
- `ImportError` - If module `drms` is not found.
- `FileNotFoundError` - If the JSOC doesn't have the magnetograms for that
time.
**Examples**:
```python
from swmfpy.web import download_magnetogram_hmi
import datetime as dt
# I am interested in the hmi vector magnetogram from 2014, 2, 18
time_mag = dt.datetime(2014, 2, 18, 10) # Around hour 10
# Calling it will download
filenames = download_magnetogram_hmi(mag_time=time_mag,
download_dir='mydir/')
# To see my list
print('The magnetograms I downloaded are:', filenames)
# You may call and ignore the file list
download_magnetogram_hmi(mag_time=time_mag, download_dir='mydir')
```
<a name=".swmfpy.web.download_magnetogram_adapt"></a>
#### download\_magnetogram\_adapt
```python
download_magnetogram_adapt(time, map_type='fixed', **kwargs)
```
This routine downloads GONG ADAPT magnetograms.
Downloads ADAPT magnetograms from ftp://gong2.nso.edu/adapt/maps/gong/
to a local directory. It will download all maps with the regex file
pattern: adapt4[0,1]3*yyyymmddhh
**Arguments**:
- `time` _datetime.datetime_ - Time in which you want the magnetogram.
- `map_type` _str_ - (default: 'fixed')
Choose either 'fixed' or 'central' for
the map type you want.
**kwargs:
- `download_dir` _str_ - (default is current dir) Relative directory
where you want the maps to be downloaded.
**Returns**:
- `str` - First unzipped filename found.
**Raises**:
- `NotADirectoryError` - If the adapt maps directory
is not found on the server.
- `ValueError` - If map_type is not recognized.
(i.e. not 'fixed' or 'central')
- `FileNotFoundError` - If maps were not found.
**Examples**:
```python
import datetime as dt
# Use datetime objects for the time
time_flare = dt.datetime(2018, 2, 12, hour=10)
swmfpy.web.download_magnetogram_adapt(time=time_flare,
map_type='central',
download_dir='./mymaps/')
```
<a name=".swmfpy.io"></a> <a name=".swmfpy.io"></a>
## swmfpy.io ## swmfpy.io
...@@ -300,6 +142,12 @@ from the GM model log. ...@@ -300,6 +142,12 @@ from the GM model log.
plt.plot(geo['times', geo['AL']) plt.plot(geo['times', geo['AL'])
``` ```
<a name=".swmfpy.tools"></a>
## swmfpy.tools
Tools to be used in swmfpy functions and classes. Some of the functions are
*hidden functions*.
<a name=".swmfpy.paramin"></a> <a name=".swmfpy.paramin"></a>
## swmfpy.paramin ## swmfpy.paramin
...@@ -319,7 +167,7 @@ Note, if you have repeat commands this will replace all the repeats. ...@@ -319,7 +167,7 @@ Note, if you have repeat commands this will replace all the repeats.
**Arguments**: **Arguments**:
- `parameters` _dict_ - Dictionary of strs with format - `parameters` _dict_ - Dictionary of strs with format
replace = {'COMMAND': ['value', 'comments', ...]} replace = 'COMMAND': ['value', 'comments', ...]
This is case sensitive. This is case sensitive.
- `input_file` _str_ - String of PARAM.in file name. - `input_file` _str_ - String of PARAM.in file name.
- `output_file` _str_ - (default 'PARAM.in') The output file to write to. - `output_file` _str_ - (default 'PARAM.in') The output file to write to.
...@@ -338,7 +186,7 @@ Note, if you have repeat commands this will replace all the repeats. ...@@ -338,7 +186,7 @@ Note, if you have repeat commands this will replace all the repeats.
**Examples**: **Examples**:
```python ```python
change['SOLARWINDFILE'] = [['T', 'UseSolarWindFile'], change['`SOLARWINDFILE`'] = [['T', 'UseSolarWindFile'],
['new_imf.dat', 'NameSolarWindFile']] ['new_imf.dat', 'NameSolarWindFile']]
# This will overwrite PARAM.in # This will overwrite PARAM.in
swmfpy.paramin.replace('PARAM.in.template', change) swmfpy.paramin.replace('PARAM.in.template', change)
...@@ -390,3 +238,167 @@ values for the parameters. ...@@ -390,3 +238,167 @@ values for the parameters.
this, try using the `num_of_values` keyword. This is helpful if your this, try using the `num_of_values` keyword. This is helpful if your
PARAM.in is comment heavy. PARAM.in is comment heavy.
<a name=".swmfpy.web"></a>
## swmfpy.web
Tools to retrieve and send data on the web.
Here are a collection of tools to work with data on the internet. Thus,
this module mostly requires an internet connection.
<a name=".swmfpy.web.get_omni_data"></a>
#### get\_omni\_data
```python
get_omni_data(time_from, time_to, **kwargs)
```
Retrieve omni solar wind data over http.
This will download omni data from https://spdf.gsfc.nasa.gov/pub/data/omni
and put it into a dictionary. If your data is large, then make a csv and
use swmfpy.io.read_omni_data().
**Arguments**:
- `time_from` _datetime.datetime_ - The start time of the solar wind
data that you want to receive.
- `time_to` _datetime.datetime_ - The end time of the solar wind data
you want to receive.
**Returns**:
- `dict` - This will be a list of *all* columns
available in the omni data set.
**Examples**:
```python
import datetime
import swmfpy.web
storm_start = datetime.datetime(year=2000, month=1, day=1)
storm_end = datetime.datetime(year=2000, month=2, day=15)
data = swmfpy.web.get_omni_data(time_from=storm_start,
time_to=storm_end)
```
<a name=".swmfpy.web.download_magnetogram_hmi"></a>
#### download\_magnetogram\_hmi
```python
download_magnetogram_hmi(mag_time, hmi_map, **kwargs)
```
Downloads HMI vector magnetogram fits files.
This will download vector magnetogram FITS files from
Joint Science Operations Center (JSOC) near a certain hour.
This unfortunately depends on sunpy and drms, if you don't have it try,
```bash
pip install -U --user sunpy drms
```
**Arguments**:
- `mag_time` _datetime.datetime_ - Time after which to find
vector magnetograms.
- `hmi_map` _str_ - JSOC prefix for hmi maps. Currently can only do
'hmi.B_720s' and 'hmi.b_synoptic.small'.
**kwargs:
- `download_dir` _str_ - Relative directory to download to.
- `verbose` _bool_ - (default False) print out the files it's downloading.
**Returns**:
- `str` - list of filenames downloaded.
**Raises**:
- `ImportError` - If module `drms` is not found.
- `FileNotFoundError` - If the JSOC doesn't have the magnetograms for that
time.
**Examples**:
```python
from swmfpy.web import download_magnetogram_hmi
import datetime as dt
# I am interested in the hmi vector magnetogram from 2014, 2, 18
time_mag = dt.datetime(2014, 2, 18, 10) # Around hour 10
# Calling it will download
filenames = download_magnetogram_hmi(mag_time=time_mag,
hmi_map='B_720s',
download_dir='mydir/')
# To see my list
print('The magnetograms I downloaded are:', filenames)
# You may call and ignore the file list
download_magnetogram_hmi(mag_time=time_mag,
hmi_map='b_synoptic_small',
download_dir='mydir')
```
<a name=".swmfpy.web.download_magnetogram_adapt"></a>
#### download\_magnetogram\_adapt
```python
download_magnetogram_adapt(time, map_type='fixed', **kwargs)
```
This routine downloads GONG ADAPT magnetograms.
Downloads ADAPT magnetograms from ftp://gong2.nso.edu/adapt/maps/gong/
to a local directory. It will download all maps with the regex file
pattern: adapt4[0,1]3*yyyymmddhh
**Arguments**:
- `time` _datetime.datetime_ - Time in which you want the magnetogram.
- `map_type` _str_ - (default: 'fixed')
Choose either 'fixed' or 'central' for
the map type you want.
**kwargs:
- `download_dir` _str_ - (default is current dir) Relative directory
where you want the maps to be downloaded.
**Returns**:
- `str` - First unzipped filename found.
**Raises**:
- `NotADirectoryError` - If the adapt maps directory
is not found on the server.
- `ValueError` - If map_type is not recognized.
(i.e. not 'fixed' or 'central')
- `FileNotFoundError` - If maps were not found.
**Examples**:
```python
import datetime as dt
# Use datetime objects for the time
time_flare = dt.datetime(2018, 2, 12, hour=10)
swmfpy.web.download_magnetogram_adapt(time=time_flare,
map_type='central',
download_dir='./mymaps/')
```
...@@ -10,7 +10,7 @@ with open('README.markdown') as readme_fh: ...@@ -10,7 +10,7 @@ with open('README.markdown') as readme_fh:
setuptools.setup( setuptools.setup(
name='swmfpy', name='swmfpy',
version='2020.0', version='2020.1',
author='Qusai Al Shidi', author='Qusai Al Shidi',
author_email='qusai@umich.edu', author_email='qusai@umich.edu',
description='''A collection of tools for the Space Weather Modelling description='''A collection of tools for the Space Weather Modelling
......
...@@ -19,7 +19,7 @@ These are not automatically imported. Might have extra dependancies. ...@@ -19,7 +19,7 @@ These are not automatically imported. Might have extra dependancies.
""" """
__author__ = 'Qusai Al Shidi' __author__ = 'Qusai Al Shidi'
__license__ = 'MIT' __license__ = 'MIT'
__version__ = '2020.0' __version__ = '2020.1'
__maintainer__ = 'Qusai Al Shidi' __maintainer__ = 'Qusai Al Shidi'
__email__ = 'qusai@umich.edu' __email__ = 'qusai@umich.edu'
......
...@@ -17,7 +17,7 @@ def replace_command(parameters, input_file, output_file='PARAM.in'): ...@@ -17,7 +17,7 @@ def replace_command(parameters, input_file, output_file='PARAM.in'):
Args: Args:
parameters (dict): Dictionary of strs with format parameters (dict): Dictionary of strs with format
replace = {'COMMAND': ['value', 'comments', ...]} replace = 'COMMAND': ['value', 'comments', ...]
This is case sensitive. This is case sensitive.
input_file (str): String of PARAM.in file name. input_file (str): String of PARAM.in file name.
output_file (str): (default 'PARAM.in') The output file to write to. output_file (str): (default 'PARAM.in') The output file to write to.
...@@ -30,7 +30,7 @@ def replace_command(parameters, input_file, output_file='PARAM.in'): ...@@ -30,7 +30,7 @@ def replace_command(parameters, input_file, output_file='PARAM.in'):
Examples: Examples:
```python ```python
change['SOLARWINDFILE'] = [['T', 'UseSolarWindFile'], change['#SOLARWINDFILE'] = [['T', 'UseSolarWindFile'],
['new_imf.dat', 'NameSolarWindFile']] ['new_imf.dat', 'NameSolarWindFile']]
# This will overwrite PARAM.in # This will overwrite PARAM.in
swmfpy.paramin.replace('PARAM.in.template', change) swmfpy.paramin.replace('PARAM.in.template', change)
......
#!/usr/bin/env python
"""Tools to be used in swmfpy functions and classes. Some of the functions are
*hidden functions*.
"""
__author__ = 'Qusai Al Shidi'
__email__ = 'qusai@umich.edu'
def _import_error_string(library):
return f'Error importing drms. Maybe try `pip install -U {library} --user ` .'
def _nearest(pivot, items):
"""Returns nearest point"""
return min(items, key=lambda x: abs(x - pivot))
...@@ -8,6 +8,7 @@ __email__ = 'qusai@umich.edu' ...@@ -8,6 +8,7 @@ __email__ = 'qusai@umich.edu'
import datetime as dt import datetime as dt
import urllib import urllib
from .tools import _import_error_string, _nearest
def get_omni_data(time_from, time_to, **kwargs): def get_omni_data(time_from, time_to, **kwargs):
...@@ -142,21 +143,24 @@ def _check_bad_omni_num(value_string): ...@@ -142,21 +143,24 @@ def _check_bad_omni_num(value_string):
return True return True
def download_magnetogram_hmi(mag_time, **kwargs):
def download_magnetogram_hmi(mag_time, hmi_map='hmi.B_720s', **kwargs):
"""Downloads HMI vector magnetogram fits files. """Downloads HMI vector magnetogram fits files.
This will download vector magnetogram FITS files from This will download vector magnetogram FITS files from
Joint Science Operations Center (JSOC) near a certain hour. Joint Science Operations Center (JSOC) near a certain hour.
This unfortunately depends on sunpy/drms, if you don't have it try, This unfortunately depends on sunpy and drms, if you don't have it try,
```bash ```bash
pip install -U --user drms pip install -U --user sunpy drms
``` ```
Args: Args:
mag_time (datetime.datetime): Time after which to find mag_time (datetime.datetime): Time after which to find
vector magnetograms. vector magnetograms.
hmi_map (str): JSOC prefix for hmi maps. Currently can only do
'hmi.B_720s' and 'hmi.b_synoptic.small'.
**kwargs: **kwargs:
download_dir (str): Relative directory to download to. download_dir (str): Relative directory to download to.
...@@ -180,13 +184,16 @@ def download_magnetogram_hmi(mag_time, **kwargs): ...@@ -180,13 +184,16 @@ def download_magnetogram_hmi(mag_time, **kwargs):
# Calling it will download # Calling it will download
filenames = download_magnetogram_hmi(mag_time=time_mag, filenames = download_magnetogram_hmi(mag_time=time_mag,
hmi_map='B_720s',
download_dir='mydir/') download_dir='mydir/')
# To see my list # To see my list
print('The magnetograms I downloaded are:', filenames) print('The magnetograms I downloaded are:', filenames)
# You may call and ignore the file list # You may call and ignore the file list
download_magnetogram_hmi(mag_time=time_mag, download_dir='mydir') download_magnetogram_hmi(mag_time=time_mag,
hmi_map='b_synoptic_small',
download_dir='mydir')
``` ```
""" """
...@@ -194,24 +201,15 @@ def download_magnetogram_hmi(mag_time, **kwargs): ...@@ -194,24 +201,15 @@ def download_magnetogram_hmi(mag_time, **kwargs):
try: try:
import drms import drms
except ImportError: except ImportError:
raise ImportError('''Error importing drms. Maybe try raise ImportError(_import_error_string('drms'))
`pip install -U --user drms` .
''')
get_urls = {
'hmi.B_720s': _get_urls_hmi_b720,
'hmi.b_synoptic_small': _get_urls_hmi_b_synoptic_small,
}
client = drms.Client() client = drms.Client()
query_string = 'hmi.B_720s['
query_string += f'{mag_time.year}.'
query_string += f'{str(mag_time.month).zfill(2)}.'
query_string += f'{str(mag_time.day).zfill(2)}_'
query_string += f'{str(mag_time.hour).zfill(2)}'
query_string += f'/1h]'
data = client.query(query_string, key='T_REC', seg='field')
times = drms.to_datetime(data[0].T_REC) urls = get_urls[hmi_map](client, mag_time)
nearest_time = _nearest(mag_time, times)
# Generator to find the nearest time
urls = ((data_time, mag_url) for (data_time, mag_url)
in zip(times, data[1].field) if data_time == nearest_time)
# Download data # Download data
if kwargs.get('verbose', False): if kwargs.get('verbose', False):
...@@ -241,6 +239,61 @@ def download_magnetogram_hmi(mag_time, **kwargs): ...@@ -241,6 +239,61 @@ def download_magnetogram_hmi(mag_time, **kwargs):
return return_name return return_name
def _get_urls_hmi_b_synoptic_small(client, mag_time):
"""Returns for #download_magnetogram_hmi needed urls
Args:
client (drms.Client): To query and return urls.
mag_time (datetime.datetime): To find nearest magnetogram.
Returns:
generator that yields (datetime.datetime, str): Time of magnetogram,
suffix url of magnetogram
"""
import drms
try:
from sunpy.coordinates.sun import carrington_rotation_number
except ImportError as error:
print(_import_error_string('sunpy'))
raise error
cr_number = int(round(carrington_rotation_number(mag_time)))
query_string = f'hmi.b_synoptic_small[{int(round(cr_number))}]'
components = ['Bp', 'Bt', 'Br']
data = client.query(query_string, seg=components)
# Generator to find the nearest time
prefix_str = 'CR' + str(cr_number) + '_' + str(mag_time)
urls = ((prefix_str, data[component][0]) for component in components)
return urls