Note: The default ITS GitLab runner is a shared resource and is subject to slowdowns during heavy usage.
You can run your own GitLab runner that is dedicated just to your group if you need to avoid processing delays.

Commit 4b4cfb52 authored by Qusai Al Shidi's avatar Qusai Al Shidi 💬
Browse files

Speed up of `get_omni_data`

parent 69033fcb
...@@ -32,9 +32,8 @@ These are automatically imported. ...@@ -32,9 +32,8 @@ These are automatically imported.
### Extra Modules ### Extra Modules
These are not automatically imported. Might have extra dependancies. - `swmfpy.tools` Tools used in swmfpy. You might find these useful but it is
unecessary.
*None yet.*
<a name=".swmfpy.write_imf_from_omni"></a> <a name=".swmfpy.write_imf_from_omni"></a>
#### write\_imf\_from\_omni #### write\_imf\_from\_omni
...@@ -88,6 +87,7 @@ use these to preprocess before submitting jobs. ...@@ -88,6 +87,7 @@ use these to preprocess before submitting jobs.
#### get\_omni\_data #### get\_omni\_data
```python ```python
@lru_cache(maxsize=4)
get_omni_data(time_from, time_to, **kwargs) get_omni_data(time_from, time_to, **kwargs)
``` ```
...@@ -97,6 +97,12 @@ This will download omni data from https://spdf.gsfc.nasa.gov/pub/data/omni ...@@ -97,6 +97,12 @@ This will download omni data from https://spdf.gsfc.nasa.gov/pub/data/omni
and put it into a dictionary. If your data is large, then make a csv and and put it into a dictionary. If your data is large, then make a csv and
use swmfpy.io.read_omni_data(). use swmfpy.io.read_omni_data().
Note that calling this more than once with the same arguments will point to
your original dictionary that it created. This is to speed up code that
calls this multiple times as it requires internet access and download.
If you mutate your original try doing an omni_dict2 = omni_dict1.copy()
and mutate the other one.
**Arguments**: **Arguments**:
- `time_from` _datetime.datetime_ - The start time of the solar wind - `time_from` _datetime.datetime_ - The start time of the solar wind
......
...@@ -22,6 +22,7 @@ $ git clone https://gitlab.umich.edu/swmf_sofware/swmfpy.git /path/to/my/dir ...@@ -22,6 +22,7 @@ $ git clone https://gitlab.umich.edu/swmf_sofware/swmfpy.git /path/to/my/dir
Then install with [pip](https://pip.pypa.io/en/stable/): Then install with [pip](https://pip.pypa.io/en/stable/):
```bash ```bash
$ pip install -U --user wheel # Might be necessary
$ pip install -U --user git+https://gitlab.umich.edu/swmf_software/swmfpy.git@master $ pip install -U --user git+https://gitlab.umich.edu/swmf_software/swmfpy.git@master
``` ```
......
...@@ -10,6 +10,7 @@ __email__ = 'qusai@umich.edu' ...@@ -10,6 +10,7 @@ __email__ = 'qusai@umich.edu'
import datetime as dt import datetime as dt
import ftplib import ftplib
from functools import lru_cache
import gzip import gzip
from operator import itemgetter from operator import itemgetter
import shutil import shutil
...@@ -122,6 +123,7 @@ OMNI_LORES_COLS = (('Bartels rotation number', 'bartels'), ...@@ -122,6 +123,7 @@ OMNI_LORES_COLS = (('Bartels rotation number', 'bartels'),
) )
@lru_cache(maxsize=4)
def get_omni_data(time_from, time_to, **kwargs): def get_omni_data(time_from, time_to, **kwargs):
"""Retrieve omni solar wind data over http. """Retrieve omni solar wind data over http.
...@@ -129,6 +131,12 @@ def get_omni_data(time_from, time_to, **kwargs): ...@@ -129,6 +131,12 @@ def get_omni_data(time_from, time_to, **kwargs):
and put it into a dictionary. If your data is large, then make a csv and and put it into a dictionary. If your data is large, then make a csv and
use swmfpy.io.read_omni_data(). use swmfpy.io.read_omni_data().
Note that calling this more than once with the same arguments will point to
your original dictionary that it created. This is to speed up code that
calls this multiple times as it requires internet access and download.
If you mutate your original try doing an omni_dict2 = omni_dict1.copy()
and mutate the other one.
Args: Args:
time_from (datetime.datetime): The start time of the solar wind time_from (datetime.datetime): The start time of the solar wind
data that you want to receive. data that you want to receive.
...@@ -251,10 +259,7 @@ def _bad_omni_num(value_string): ...@@ -251,10 +259,7 @@ def _bad_omni_num(value_string):
"""Returns true if bad or false if not. Bad numbers usually just have 9s """Returns true if bad or false if not. Bad numbers usually just have 9s
in omni. in omni.
""" """
for char in value_string: return all([char in ('9', '.') for char in value_string])
if char not in ('9', '.'):
return False
return True
def download_magnetogram_hmi(mag_time, hmi_map='hmi.B_720s', **kwargs): def download_magnetogram_hmi(mag_time, hmi_map='hmi.B_720s', **kwargs):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment