Note: The default ITS GitLab runner is a shared resource and is subject to slowdowns during heavy usage.
You can run your own GitLab runner that is dedicated just to your group if you need to avoid processing delays.

Commit 0e2d5042 authored by Qusai Al Shidi's avatar Qusai Al Shidi 💬
Browse files

Initial script commit looks like it's working

parent c546d3c0
- bb06e1c **(2020-05-20, Qusai Al Shidi)**: _CHEAD -> master, tag: v2020.2, origin/master, origin/HEAD_: v2020.2 changelog
- c10ece8 **(2020-05-20, Qusai Al Shidi)**: _C_: v2020.2 commit
- 9cb028f **(2020-05-20, Qusai Al Shidi)**: _C_: Fixed installation directions
- 48200d7 **(2020-05-20, Qusai Al Shidi)**: _C_: Merge branch 'imf-f107' into 'master'
- 3c75787 **(2020-05-19, Qusai Al Shidi)**: _Corigin/imf-f107, imf-f107_: Working and tested. Made documentation. Added new feature swmfpy.write_imf_from_omni
- e4f879c **(2020-05-13, Qusai Al Shidi)**: _C_: Transformation function made. Write IMF untested. Almost done.
- 0d243de **(2020-05-08, Qusai Al Shidi)**: _C_: Not a working commit. Just pushing this to work on another computer
- 6784109 **(2020-05-04, Qusai Al Shidi)**: _C_: Nicer omni key names. original_colnames kwarg added if you have code that depended on the old names
- f564dbe **(2020-05-04, Qusai Al Shidi)**: _C_: Merge branch 'master' into imf-f107
- d4a5c35 **(2020-05-04, Qusai Al Shidi)**: _C_: Removed -install messages Installation section still needs work
- 41efc0b **(2020-05-04, Qusai Al Shidi)**: _C_: Escape of # unecessary in markdown
- 63f6dab **(2020-05-04, Qusai Al Shidi)**: _C_: lint fixes and found documentation method to show # symbol. Use `share/`
- 8038562 **(2020-04-29, Qusai Al Shidi)**: _C_: Perhaps this change in contributing will show the youtube video.
- 3cbe318 **(2020-04-29, Qusai Al Shidi)**: _C_: Added logo to readme and contributing should show the trunk based git development vid
- b7ed409 **(2020-04-29, Qusai Al Shidi)**: _C_: Added logo
- 339c191 **(2020-04-29, Qusai Al Shidi)**: _C_: Untested initial commit for the new IMF and F10.7 features needed for SWMF GM
- 4067a38 **(2020-04-28, Qusai Al Shidi)**: _C_: Added changelog
- 8acb8c3 **(2020-04-28, Qusai Al Shidi)**: _Ctag: v2020.1_: Version 2020.1. Changed download_magnetogram_hmi to take a new argument for either B_720s or b_synoptic_small which is needed.
- 3a10227 **(2020-04-24, Qusai Al Shidi)**: _Ctag: v2020.0_: 2020.0 commit
- 6f8cbfc **(2020-04-21, Qusai Al Shidi)**: _C_: Updated readme to reflect the new state of swmfpy in SWMF installation
- 9141aa9 **(2020-04-21, Qusai Al Shidi)**: _C_: commit
- 6ab869c **(2020-04-21, Qusai Al Shidi)**: _C_: Added tests directory but no tests yet. added to ensure it can be installed
- 37d5b4f **(2020-04-19, Gabor Toth)**: _C_: Update README.markdown
- 6c9c58a **(2020-04-08, Qusai Al Shidi)**: _C_: Documentation fixes
- dffef34 **(2020-04-08, Qusai Al Shidi)**: _C_: Removed deprecated functions and added to documentation
- 8011884 **(2020-04-08, Qusai Al Shidi)**: _C_: Removed deprecated functions. Will write new ones.
- 5e00d81 **(2020-04-08, Qusai Al Shidi)**: _C_: Merge branch 'download_magnetogram_hmi' into 'master'
- 7518f1e **(2020-04-08, Qusai Al Shidi)**: _Corigin/download_magnetogram_hmi, download_magnetogram_hmi_: Fixed download_magnetogram_adapt in filename return. Now download_magnetogram_hmi downloads only the nearest time. hmi_ is preppended to filename.
- 523f958 **(2020-04-03, Qusai Al Shidi)**: _C_: Fixed download_magnetogram_adapt. Also raise new error if hmi mag not found
- 36ba08e **(2020-04-03, Qusai Al Shidi)**: _C_: Added section to CONTRIBUTING
- 39ba725 **(2020-04-03, Qusai Al Shidi)**: _C_: Fixed docstring
- c3b6429 **(2020-04-03, Qusai Al Shidi)**: _C_: Working download of hmi magnetograms
- 5fb291b **(2020-04-02, Qusai Al Shidi)**: _C_: Fixed get_omni_data to get all months instead of months-1
- ee68b59 **(2020-03-31, Qusai Al Shidi)**: _C_: Documentation with toc
- 3ef8ce1 **(2020-03-31, Qusai Al Shidi)**: _C_: More doc fixes
- b5a80a1 **(2020-03-31, Qusai Al Shidi)**: _C_: Hopefully this works for docs
- b5760f3 **(2020-03-31, Qusai Al Shidi)**: _C_: More doc fixes
- 3b96ad4 **(2020-03-31, Qusai Al Shidi)**: _C_: Oops documetnation fixed
- 76a1e2c **(2020-03-31, Qusai Al Shidi)**: _C_: Documentation fixes
- 1a7f0c7 **(2020-03-31, Qusai Al Shidi)**: _C_: Autodoc
- bfefaee **(2020-03-30, Qusai Al Shidi)**: _C_: Minor change that doesn't matter
- 57ac926 **(2020-03-26, Qusai Al Shidi)**: _C_: Ensure text file opening of wdc
- 7f32fdd **(2020-03-20, Qusai Al Shidi)**: _C_: html documentation through pydoc
- f53e5c5 **(2020-03-20, Qusai Al Shidi)**: _C_: Added documentation improvements
- cc46626 **(2020-03-20, Qusai Al Shidi)**: _C_: Improved pydoc
- f9f3b69 **(2020-03-20, Qusai Al Shidi)**: _C_: Improved paramin functions
- 113cbec **(2020-03-20, Qusai Al Shidi)**: _C_: Some paramin changes untested
- a7e3222 **(2020-03-17, Qusai Al Shidi)**: _C_: Many changes to io for better parsing. WDC functions work better
- 26a66ca **(2020-03-13, Qusai Al Shidi)**: _C_: Fixed WDC functions in
- cb2f765 **(2020-03-12, Qusai Al Shidi)**: _C_: COVID commit again
- 430569d **(2020-03-11, Qusai Al Shidi)**: _C_: Commit just in case CSRB closes due to coronavirus
- 505679c **(2020-03-11, Qusai Al Shidi)**: _C_: io.read_gm_log does not rely on pandas anymore instead it gives a dict
- 7a4e7d2 **(2020-03-10, Qusai Al Shidi)**: _C_: Made dependency handling more clear.
- 2d6c838 **(2020-03-10, Qusai Al Shidi)**: _C_: Merging my adapt branch
- 63fbd1a **(2020-03-10, Qusai Al Shidi)**: _C_: removed code since it's deprecated
- 3f9a436 **(2020-03-09, Qusai Al Shidi)**: _C_: Merge from adapt branch
- 567bea8 **(2020-03-09, Qusai Al Shidi)**: _C_: Working on this
- 7512947 **(2020-03-06, Qusai Al Shidi)**: _C_: Hid coarse_filtering from import
- 52aaf23 **(2020-03-04, Qusai Al Shidi)**: _C_: Cleaning Zhenguang code further.
- 36cec6b **(2020-03-04, Qusai Al Shidi)**: _C_: Added cleaned Zhenguang code.
- 063e33c **(2020-03-03, Qusai Al Shidi)**: _C_: Fixed paramin replace and changed name added paramin.read_command()
- 569080e **(2020-03-03, Qusai Al Shidi)**: _C_: Help page of swmfpy made clearer.
- 79d6122 **(2020-03-03, Qusai Al Shidi)**: _C_: Better module docstrings
- 5b3c7cb **(2020-03-03, Qusai Al Shidi)**: _C_: Fixed swmfpy module docstring to be better
- 45de9a4 **(2020-03-03, Qusai Al Shidi)**: _C_: Changed name of to Removed DOCUMENTATION link because it is not quite ready. Readability changes.
- 4c4abad **(2020-03-02, Qusai Al Shidi)**: _C_: Fixed PEP-8 of get_omni_data, fixed CONTRIBUTING
- 41dc03e **(2020-03-02, Qusai Al Shidi)**: _C_: Made swmfpy.spdf.get_omni_data faster
- 519991f **(2020-03-02, Qusai Al Shidi)**: _C_: Added swmf.spdf.get_omni_data that downloads omni data directly into a dictionary
- a02018f **(2020-02-27, Qusai Al Shidi)**: _C_: changed name to author in contributing
- 6ccdbcd **(2020-02-27, Qusai Al Shidi)**: _C_: Changed contributing to include metadata directions
- fd26dd4 **(2020-02-27, Qusai Al Shidi)**: _C_: Fixed Documentation
- b1af5a3 **(2020-02-27, Qusai Al Shidi)**: _C_: Documentation needed fix
- 02be787 **(2020-02-27, Qusai Al Shidi)**: _C_: Added pdoc generated documentation
- d07a704 **(2020-02-26, Qusai Al Shidi)**: _C_: Added contributing
- fa37f91 **(2020-02-26, Qusai Al Shidi)**: _C_: Fixed readability in read_wdc_ae docstring
- a1ad542 **(2020-02-20, Qusai Al Shidi)**: _C_: gitlab doesn't allow html docs will host externally
- 65aa496 **(2020-02-20, Qusai Al Shidi)**: _C_: Web docs
- cf4b705 **(2020-02-20, Qusai Al Shidi)**: _C_: Added web docs
- 19c2fd5 **(2020-02-19, Qusai Al Shidi)**: _C_: more readme fix
- 7d24114 **(2020-02-19, Qusai Al Shidi)**: _C_: Merge branch 'master' of
- f10e0ff **(2020-02-19, Qusai Al Shidi)**: _C_: Added better clone line because of file structure change
- b0ca0ba **(2020-02-19, Qusai Al Shidi)**: _C_: Added better clone line because of file structure change
- dfee4c9 **(2020-02-19, Qusai Al Shidi)**: _C_: Restructured file structure
- 18ebfe2 **(2020-02-19, Qusai Al Shidi)**: _C_: File structure changes
- a08b83e **(2020-02-19, Qusai Al Shidi)**: _C_: paramin_replace usable
- 3d2f2e0 **(2020-02-17, Qusai Al Shidi)**: _C_: Changed module docstring
- e0306b6 **(2020-02-17, Qusai Al Shidi)**: _C_: replace paramin function works as expected now
- f68d57f **(2020-02-17, Qusai Al Shidi)**: _C_: Fixed syntax highlighting
- 281329c **(2020-02-17, Qusai Al Shidi)**: _C_: Added more readme
- be28064 **(2020-02-17, Qusai Al Shidi)**: _C_: Commit before gitlab transfer haven't tested new replace features
- f0575f2 **(2020-02-07, Qusai Al Shidi)**: _C_: PARAMin find replace added
- cf39804 **(2019-11-27, Qusai Al Shidi)**: _C_: Reads gm logs well
- c2c47c0 **(2019-11-18, Qusai Al Shidi)**: _C_: Initial commit for library
- 8b6ace2 **(2019-10-17, Qusai Al Shidi)**: _C_: Added radiation belt model input file creation as well.
- a2ce534 **(2019-08-30, Qusai Al Shidi)**: _C_: Just have the python header.
- f4c7b4f **(2019-08-30, Qusai Al Shidi)**: _C_: Fixed microsecond to millisecond mistake in write_data
- 0943f8d **(2019-08-28, Qusai Al Shidi)**: _C_: Updated readme
- eb4e761 **(2019-08-28, Qusai Al Shidi)**: _C_: Added a way to write out DataFrame into IMF.dat file
- 41a6385 **(2019-08-28, Qusai Al Shidi)**: _C_: Added function combine data for ACE and WIND data that interpolates
- ab0813c **(2019-08-22, Qusai Al Shidi)**: _C_: Added fix if data ends with baddata
- 6d1bbb4 **(2019-08-20, Qusai Al Shidi)**: _C_: Added readme
- 0b11a57 **(2019-08-20, Qusai Al Shidi)**: _C_: Merge branch 'master' of
- 864495e **(2019-08-20, Qusai Al Shidi)**: _C_: Commit before github push
- e753deb **(2019-08-20, Qusai Al Shidi)**: _C_: Initial commit
- 2c71470 **(2019-08-19, Qusai Al Shidi)**: _C_: Added closing of files
- 2f84fb6 **(2019-08-19, Qusai Al Shidi)**: _C_: Working scripts
- 4500bca **(2019-08-19, Qusai Al Shidi)**: _C_: Initial commit.
\ No newline at end of file
Before submitting pull requests please make sure your code complies with the following.
Git etiquette
If you're planning on adding a feature (new function or class) then create your own branch,
$ git checkout -b my_new_feature
and start commiting there to test and work on. We will follow a [trunk-based development model]( This means we will rapidly merge branches once you have something stable and continue with master branch. So make sure to push something stable instead of being in your feature branch for too long. I will try to address your pull request ASAP.
[![Trunk-based Development](]( "Trunk-based Development")
Coding Style: PEP 8
The style for your code must follow the [PEP 8]( style guide. It would be helpful to use a linter like [pylint]( to check whether your code is complying before submitting.
$ pylint
Coding Standard
Please write readable code. Make sure your function names well describes your functions. Always include docstrings that use the [Google coding style]( The Google coding style guide is a good document to follow so I recommend reading it.
Include metadata if you made a new function or module. Name and email will suffice. Use the variable names `__author__` and `__email__` for modules and comments for functions that you add to someone else's module. For example:
"""A new package for swmfpy
__all__ = ['funcs', 'to', 'export']
__author__ = 'Rita Hayworth'
__email__ = ''
Or for a function:
def my_new_func(some_args):
"""my docstring
# Author: Diane Selwyn
# Email:
# function body
If your code has dependencies that is large and uncommon (e.g. SpacePy) then please import it in the function and be explicit about the dependencies. For example you may create a function:
def func_with_dependency():
"""Func does this.
Depends on spacepy. Try `pip install -U spacepy`
import spacepy
# Function body
Then your functions will be in swmfpy.spacepy which has its own namespace.
If you want to create a new module make sure what you're trying to do can't just fit in the modules already made.
# Table of Contents
* [swmfpy](#.swmfpy)
* [write\_imf\_from\_omni](#.swmfpy.write_imf_from_omni)
* [swmfpy.web](#.swmfpy.web)
* [get\_omni\_data](#.swmfpy.web.get_omni_data)
* [download\_magnetogram\_hmi](#.swmfpy.web.download_magnetogram_hmi)
* [download\_magnetogram\_adapt](#.swmfpy.web.download_magnetogram_adapt)
* [](
* [write\_imf\_input](
* [read\_wdc\_ae](
* [read\_wdc\_asy\_sym](
* [read\_gm\_log](
* [swmfpy.paramin](#.swmfpy.paramin)
* [replace\_command](#.swmfpy.paramin.replace_command)
* [read\_command](#.swmfpy.paramin.read_command)
* [](
<a name=".swmfpy"></a>
## swmfpy
A collection of tools to read, write, visualize with the
Space Weather Modeling Framework (SWMF).
### Modules
These are automatically imported.
- `` Input/Output tools.
- `swmfpy.paramin` editing tools.
- `swmfpy.web` Internet data downloading/uploading tools.
### Extra Modules
These are not automatically imported. Might have extra dependancies.
*None yet.*
<a name=".swmfpy.write_imf_from_omni"></a>
#### write\_imf\_from\_omni
write_imf_from_omni(time_from, time_to, filename='IMF.dat', **kwargs)
Writes an IMF.dat file for the geospace model runs for a specific time
- `time_from` _datetime.datetime_ - Time to begin omni data retrieval
- `time_to` _datetime.datetime_ - Time to end omni data retrieval
- `filename` _str_ - The filename for the dat file, defaults to 'IMF.dat'.
see `` and `swmfpy.web.get_omni_data()`
Using this function is simple:
import swmfpy
import datetime as dt
times = (dt.datetime(2014, 2, 2), dt.datetime(2014, 2, 4))
# Usually the kwargs are unecessary
# Sometimes this
swmfpy.write_imf_input(*times, filename='run/IMF.dat')
<a name=".swmfpy.web"></a>
## swmfpy.web
### Tools to download/upload data on the web
Here are a collection of tools to work with data on the internet. Thus,
this module mostly requires an internet connection. Which on some
supercomputers would be turned off during a job run. In scripts, make sure to
use these to preprocess before submitting jobs.
<a name=".swmfpy.web.get_omni_data"></a>
#### get\_omni\_data
get_omni_data(time_from, time_to, **kwargs)
Retrieve omni solar wind data over http.
This will download omni data from
and put it into a dictionary. If your data is large, then make a csv and
- `time_from` _datetime.datetime_ - The start time of the solar wind
data that you want to receive.
- `time_to` _datetime.datetime_ - The end time of the solar wind data
you want to receive.
- `original_colnames` _bool_ - Use the original column names from the
spdf specification. The alternative is
nicer and shorter names. Defaults to
- `resolution` _str_ - (default: 'high') Here you can choose 'high' or
'low' resolution omni data. Some columns appear
in one but not the other.
- `dict` - This will be a list of *all* columns
available in the omni data set.
import datetime
import swmfpy.web
storm_start = datetime.datetime(year=2000, month=1, day=1)
storm_end = datetime.datetime(year=2000, month=2, day=15)
data = swmfpy.web.get_omni_data(time_from=storm_start,
# or for low res
data = swmfpy.web.get_omni_data(time_from=storm_start,
<a name=".swmfpy.web.download_magnetogram_hmi"></a>
#### download\_magnetogram\_hmi
download_magnetogram_hmi(mag_time, hmi_map='hmi.B_720s', **kwargs)
Downloads HMI vector magnetogram fits files.
This will download vector magnetogram FITS files from
Joint Science Operations Center (JSOC) near a certain hour.
This unfortunately depends on sunpy and drms, if you don't have it try,
pip install -U --user sunpy drms
- `mag_time` _datetime.datetime_ - Time after which to find
vector magnetograms.
- `hmi_map` _str_ - JSOC prefix for hmi maps. Currently can only do
'hmi.B_720s' and 'hmi.b_synoptic.small'.
- `download_dir` _str_ - Relative directory to download to.
- `verbose` _bool_ - (default False) print out the files it's downloading.
- `str` - list of filenames downloaded.
- `ImportError` - If module `drms` is not found.
- `FileNotFoundError` - If the JSOC doesn't have the magnetograms for that
from swmfpy.web import download_magnetogram_hmi
import datetime as dt
# I am interested in the hmi vector magnetogram from 2014, 2, 18
time_mag = dt.datetime(2014, 2, 18, 10) # Around hour 10
# Calling it will download
filenames = download_magnetogram_hmi(mag_time=time_mag,
# To see my list
print('The magnetograms I downloaded are:', filenames)
# You may call and ignore the file list
<a name=".swmfpy.web.download_magnetogram_adapt"></a>
#### download\_magnetogram\_adapt
download_magnetogram_adapt(time, map_type='fixed', **kwargs)
This routine downloads GONG ADAPT magnetograms.
Downloads ADAPT magnetograms from
to a local directory. It will download all maps with the regex file
pattern: adapt4[0,1]3*yyyymmddhh
- `time` _datetime.datetime_ - Time in which you want the magnetogram.
- `map_type` _str_ - (default: 'fixed')
Choose either 'fixed' or 'central' for
the map type you want.
- `download_dir` _str_ - (default is current dir) Relative directory
where you want the maps to be downloaded.
- `str` - First unzipped filename found.
- `NotADirectoryError` - If the adapt maps directory
is not found on the server.
- `ValueError` - If map_type is not recognized.
(i.e. not 'fixed' or 'central')
- `FileNotFoundError` - If maps were not found.
import datetime as dt
# Use datetime objects for the time
time_flare = dt.datetime(2018, 2, 12, hour=10)
<a name=""></a>
### Input/Output tools
Here are tools to read and write files relating to SWMF.
<a name=""></a>
#### write\_imf\_input
write_imf_input(imf_data, filename='IMF.dat', **kwargs)
Creates the IMF.dat input file for the SWMF BATS-R-US geospace model.
`imf_data` must be a dictionary of array_like objects with same length
in data. In swmfpy Pythonic versions are always preferred so the 'times'
must be `datetime.datetime` array.
imf_data = dict(times, bx, by, bz, vx, vy, vz, density, temperature)
- `imf_data` _dict_ - This dictionary contains the solar wind data.
- `filename` _str_ - (default: 'IMF.dat') Filename to write to.
- `commands` _[str]_ - (default: None) List of commands to write to imf
input file (indexed by line then by tabs on same
line). *Note*: Must be a list if have one command
- `TypeError` - If commands is not a list or tuple. It must be at least a
one element list of strings.
- `AssertionError` - If inputs aren't prepared properly (key names)
from import write_imf_input
# Prepare imf dictionary: imf_data
write_imf_input(imf_data, filename='run/IMF.dat')
<a name=""></a>
#### read\_wdc\_ae
Read an auroral electrojet (AE) indeces from Kyoto's World Data Center
text file into a dictionary of lists.
- `wdc_filename` _str_ - Filename of wdc data from
- `dict` - Auroral indeces 'AL', 'AE', 'AO', 'AU'
- `datetime.datetime` - 'times'
- `int` - 'values'
<a name=""></a>
#### read\_wdc\_asy\_sym
Reads a WDC file for ASY/SYM data.
Reads an ASY/SYM file downloaded from
and puts it into a dictionary.
- `wdc_filename` _str_ - Relative filename (or file handle no.) to read.
- `dict` - of values. {'[ASY-SYM]-[D-H]': 'times': [], 'values': []}
indeces ='wdc.dat')
# Plot data
label='SYM-H [nT]'
plt.xlabel('Time [UT]')
Important to note if there is bad data it will be filled as None.
<a name=""></a>
#### read\_gm\_log
read_gm_log(filename, colnames=None, dtypes=None, index_time=True)
Make a dictionary out of the indeces outputted
from the GM model log.
- `filename` _str_ - The relative filename as a string. (or file handle no.)
- `colnames` _[str]_ - (default: None) Supply the name of the columns.
If None it will use second line
of log file.
- `dtypes` _[types]_ - (default: None) Provide types for the columns, if
None then all will be float.
- `index_time` _bool_ - (default: True) Make a column of dt.datetime objects
in dictionary key 'Time [UT]'.
- `dict` - Dictionary of the log file
To plot AL and Dst get the log files
geo ='run/GM/IO2/geoindex_e20140215-100500.log')
dst ='run/GM/IO2/log_e20140215-100500.log')
# Plot AL indeces
plt.plot(geo['times', geo['AL'])
<a name=".swmfpy.paramin"></a>
## swmfpy.paramin
### Editing files
These tools are to help script the editing and reading of files.
<a name=".swmfpy.paramin.replace_command"></a>
#### replace\_command
replace_command(parameters, input_file, output_file='')
Replace values for the parameters in a file.
Note, if you have repeat commands this will replace all the repeats.
- `parameters` _dict_ - Dictionary of strs with format
replace = '#COMMAND': ['value', 'comments', ...]
This is case sensitive.
- `input_file` _str_ - String of file name.
- `output_file` _str_ - (default '') The output file to write to.
A value of None will not output a file.
A list of lines of the file that would be outputted.
- `TypeError` - If a value given couldn't be converted to string.
change['#SOLARWINDFILE'] = [['T', 'UseSolarWindFile'],
['new_imf.dat', 'NameSolarWindFile']]
# This will overwrite
swmfpy.paramin.replace('', change)
<a name=".swmfpy.paramin.read_command"></a>
#### read\_command