Merge branch 'master' into tecplot-interpolate

.gitlab-ci.yml

+before_script:
+    - apt-get update -qy
+    - apt-get install -y python3 python3-numpy python3-pip
+    - python3 -m pip install pytest
+    - python3 -m pip install -r requirements.txt
+    - python3 -m pip install drms
+
+run-test:
+    script:
+        - python3 -m pytest

CHANGELOG.markdown

@@ -5,15 +5,24 @@ Changelog:
 Contributors:
 -------------
 
-   131	Qusai Al Shidi
+   137	Qusai Al Shidi
      2	Camilla D. K. Harris
-     1	Gabor Toth
+     2	Gabor Toth
+     1	Camilla Dodge Koslowsky Harris
      1	TKeebler
 
 Changes:
 --------
 
-- 05ac345 **(2020-06-19, Qusai Al Shidi)**: _C:HEAD -> master, origin/master, origin/HEAD:_ merged amendment :s
+- 23c6e26 **(2020-08-05, Gabor Toth)**: _C:HEAD -> carrington_rotation:_ Carrington Rotation function added and sunpy dependency removed
+- 8e18ac8 **(2020-08-04, Qusai Al Shidi)**: _C:origin/master, origin/HEAD, master:_ Fixed LaTeX citation string
+- 8211554 **(2020-08-04, Qusai Al Shidi)**: _C::_ Fixed install swmfpy line
+- f1724e7 **(2020-08-04, Qusai Al Shidi)**: _C::_ Added instructions for python venv and also snippets to contributing page.
+- d0a8308 **(2020-07-06, Qusai Al Shidi)**: _C:pybatstools:_ Fixed up `CONTRIBUTING.markdown` to be better and added issues guidelines
+- 9549026 **(2020-07-06, Qusai Al Shidi)**: _C::_ Should be `shell` and not `bash` for markdown language alias
+- 178dd8e **(2020-06-22, Qusai Al Shidi)**: _C::_ Merge branch 'tecplot-doc-update' into 'master'
+- facad25 **(2020-06-22, Camilla Dodge Koslowsky Harris)**: _C::_ Tecplot documentation update
+- 05ac345 **(2020-06-19, Qusai Al Shidi)**: _C::_ merged amendment :s
 - 8d04c39 **(2020-06-19, Qusai Al Shidi)**: _C::_ optional requires `tecplot`
 - 3e676fa **(2020-06-19, Qusai Al Shidi)**: _C::_ optional requires `tecplot`
 - 991a3a2 **(2020-06-19, Qusai Al Shidi)**: _C:tag: v2020.5:_ v2020.5 commit

CONTRIBUTING.markdown

@@ -76,6 +76,12 @@ def my_new_func(some_args):
     # function body
 ```
 
+There are helpful snippets for these:
+
+- function $91
+- class $92
+- file $93
+
 Dependencies
 ------------
 

DOCUMENTATION.markdown

@@ -15,6 +15,8 @@
   * [replace\_command](#.swmfpy.paramin.replace_command)
   * [read\_command](#.swmfpy.paramin.read_command)
 * [swmfpy.tools](#.swmfpy.tools)
+  * [interp\_nans](#.swmfpy.tools.interp_nans)
+  * [carrington\_rotation\_number](#.swmfpy.tools.carrington_rotation_number)
 * [swmfpy.tecplottools](#.swmfpy.tecplottools)
   * [apply\_equations](#.swmfpy.tecplottools.apply_equations)
 
@@ -157,13 +159,6 @@ Downloads HMI vector magnetogram fits files.
 This will download vector magnetogram FITS files from
 Joint Science Operations Center (JSOC) near a certain hour.
 
-This unfortunately depends on sunpy and drms, if you don't have it try,
-
-
-```bash
-pip install -U --user sunpy drms
-```
-
 **Arguments**:
 
 - `mag_time` _datetime.datetime_ - Time after which to find
@@ -506,6 +501,46 @@ values for the parameters.
 Tools to be used in swmfpy functions and classes. Some of the functions are
 *hidden functions*.
 
+<a name=".swmfpy.tools.interp_nans"></a>
+#### interp\_nans
+
+```python
+interp_nans(x_vals, y_vals)
+```
+
+Returns a numpy array with NaNs interpolated.
+
+**Arguments**:
+
+  x_vals (np.array):
+  x values to interpolate.
+  y_vals (np.array):
+  y values including NaNs.
+  
+
+**Returns**:
+
+- `(np.array)` - numpy array with NaNs interpolated.
+
+<a name=".swmfpy.tools.carrington_rotation_number"></a>
+#### carrington\_rotation\_number
+
+```python
+carrington_rotation_number(the_time='now')
+```
+
+Returns the carrington rotation
+
+**Arguments**:
+
+- `the_time` _datetime.datetime/str_ - The time to convert to carrington
+  rotation.
+  
+
+**Returns**:
+
+- `(int)` - Carrington Rotation
+
 <a name=".swmfpy.tecplottools"></a>
 ## swmfpy.tecplottools
 

LICENSE

-MIT License
-
-Copyright (c) 2019 Qusai Al Shidi
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+
+                   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.

README.markdown

@@ -10,10 +10,12 @@ This is a work in progress.
 Installation
 ------------
 
-Clone into the directory you want to use it.
-
 *Note*: swmfpy also is part of the SWMF and gets cloned into `SWMF/share/Python`. However, if you would like to [develop](CONTRIBUTING.markdown) for swmfpy make a clone and work that way and make a merge request.
 
+There are two methods of installing swmfpy with with a virtual environment and without a virtual environment. Only use the virtual environment if your current environment is giving you trouble.
+
+### Without Python venv
+
 Install with [pip](https://pip.pypa.io/en/stable/):
 
 ```shell
@@ -29,6 +31,45 @@ Then import it into your python project.
 import swmfpy
 ```
 
+### With Python venv
+
+Use this method if the above method is giving you trouble.
+
+Set up a python 3 virtual environment:
+
+```bash
+$ python3 -m venv ~/.venv
+```
+
+This is important, make sure that it is in your `.profile` or `.bash_profile`:
+
+```bash
+$ echo "source ~/.venv/bin/activate" >> ~/.profile
+```
+
+*Note*: You might need to use `activate.csh` instead if using `csh` and `activate.fish` in your `~/.config/fish/config.fish` instead if using `fish` shell. You are most likely using `bash` so no need to worry.
+
+Next install the software. This will take a long time as you will be compiling `numpy` from scratch.
+
+```bash
+$ source ~/.venv/bin/activate
+$ python3 -m pip install --user wheel -vvv
+$ python3 -m pip install --user cython -vvv
+$ python3 -m pip install --user git+https://gitlab.umich.edu/swmf_software/swmfpy.git@master
+```
+
+If you are using `tmux` or `GNU Screen` on a supercomputer you can safely detach your session and power off your computer and come back another time.
+
+This should be fully installed now. You should be able to import:
+
+```python
+import swmfpy
+```
+
+### Troubleshooting
+
+If you have followed these carefully and still not been able to install please submit an Issue.
+
 Documentation
 -------------
 
@@ -46,3 +87,18 @@ Issues
 ------
 
 If you are experiencing any issues or bugs please go to the [Issues](https://gitlab.umich.edu/swmf_software/swmfpy/issues) page and create an issue. Make sure you include steps to recreate the problem in your post.
+
+How to cite
+-----------
+
+You can cite this software on LaTeX like this:
+
+```latex
+@software{swmfpy,
+  author = {{Al Shidi, Qusai}},
+  title = {swmfpy},
+  url = {https://gitlab.umich.edu/swmf_software/swmfpy},
+  version = {2020.5},
+  date = {2020-06-19},
+}
+```

requirements.txt

 python-dateutil
 numpy
-drms
-sunpy

setup.py

@@ -13,7 +13,7 @@ with open('requirements.txt') as fh_requirements:
 
 setuptools.setup(
     name='swmfpy',
-    version='2020.5',
+    version='2020.6',
     author='Qusai Al Shidi',
     author_email='qusai@umich.edu',
     description='''A collection of tools for the Space Weather Modelling
@@ -31,5 +31,6 @@ setuptools.setup(
     python_requires='>=3.6',
     extras_require={
         "tecplottools": "tecplot",
+        "hmi": "drms",
         },
     )

swmfpy/__init__.py

@@ -18,7 +18,7 @@ These are automatically imported.
 """
 __author__ = 'Qusai Al Shidi'
 __license__ = 'MIT'
-__version__ = '2020.5'
+__version__ = '2020.6'
 __maintainer__ = 'Qusai Al Shidi'
 __email__ = 'qusai@umich.edu'
 

swmfpy/io.py

@@ -4,6 +4,7 @@ Here are tools to read and write files relating to SWMF.
 """
 
 import datetime as dt
+import warnings
 import numpy as np
 from .tools import _make_line
 
@@ -287,7 +288,13 @@ def read_gm_log(filename, colnames=None, dtypes=None, index_time=True):
                     if dtypes:
                         data = dtypes[col](data)
                     else:
-                        data = float(data)
+                        try:
+                            data = float(data)
+                        except ValueError as err:
+                            data = np.nan
+                            warnings.warn('Value error and no manual dtype set'
+                                          + ' changing to NaN',
+                                          RuntimeWarning)
                     return_data[colnames[col]].append(data)
 
         # datetime index

swmfpy/tools.py

@@ -5,6 +5,43 @@
 __author__ = 'Qusai Al Shidi'
 __email__ = 'qusai@umich.edu'
 
+import datetime as dt
+import numpy as np
+
+
+def interp_nans(x_vals, y_vals):
+    """Returns a numpy array with NaNs interpolated.
+
+    Args:
+        x_vals (np.array):
+            x values to interpolate.
+        y_vals (np.array):
+            y values including NaNs.
+
+    Returns:
+        (np.array): numpy array with NaNs interpolated.
+    """
+
+    nonans = np.nonzero(np.isnan(y_vals) == 0)
+    return np.interp(x_vals, x_vals[nonans], y_vals[nonans])
+
+
+def carrington_rotation_number(the_time='now'):
+    """Returns the carrington rotation
+
+    Args:
+        the_time (datetime.datetime/str): The time to convert to carrington
+                                          rotation.
+
+    Returns:
+        (int): Carrington Rotation
+    """
+    if the_time == 'now':
+        return carrington_rotation_number(dt.datetime.now())
+    if isinstance(the_time, str):
+        return carrington_rotation_number(dt.datetime.fromisoformat(the_time))
+    return int((the_time.toordinal() - 676715.2247)/27.2753)
+
 
 def _import_error_string(library):
     return (

swmfpy/web.py

@@ -13,13 +13,14 @@ import ftplib
 from functools import lru_cache
 import gzip
 from operator import itemgetter
+import os.path
 import shutil
 import urllib
+import urllib.request
+import warnings
 from dateutil import rrule
-import drms
 import numpy as np
-from sunpy.coordinates.sun import carrington_rotation_number
-from .tools import _nearest
+from .tools import _nearest, carrington_rotation_number
 
 # Global defines
 # This is straight from the format guide on spdf with nicer names as second col
@@ -268,12 +269,6 @@ def download_magnetogram_hmi(mag_time, hmi_map='hmi.B_720s', **kwargs):
     This will download vector magnetogram FITS files from
     Joint Science Operations Center (JSOC) near a certain hour.
 
-    This unfortunately depends on sunpy and drms, if you don't have it try,
-
-    ```bash
-    pip install -U --user sunpy drms
-    ```
-
     Args:
         mag_time (datetime.datetime): Time after which to find
                                       vector magnetograms.
@@ -314,6 +309,7 @@ def download_magnetogram_hmi(mag_time, hmi_map='hmi.B_720s', **kwargs):
                                  download_dir='mydir')
         ```
     """
+    import drms
 
     get_urls = {
         'hmi.B_720s': _get_urls_hmi_b720,
@@ -362,9 +358,10 @@ def _get_urls_hmi_b_synoptic_small(client, mag_time):
         generator that yields (datetime.datetime, str): Time of magnetogram,
             suffix url of magnetogram
     """
+    import drms
 
-    cr_number = int(round(carrington_rotation_number(mag_time)))
-    query_string = f'hmi.b_synoptic_small[{int(round(cr_number))}]'
+    cr_number = carrington_rotation_number(mag_time)
+    query_string = f'hmi.b_synoptic_small[{cr_number}]'
     components = ['Bp', 'Bt', 'Br']
     data = client.query(query_string, seg=components)
     # Generator to find the nearest time
@@ -384,6 +381,7 @@ def _get_urls_hmi_b720(client, mag_time):
         generator that yields (datetime.datetime, str): Time of magnetogram,
             suffix url of magnetogram
     """
+    import drms
     query_string = 'hmi.B_720s'
     query_string += f'[{mag_time.year}.'
     query_string += f'{str(mag_time.month).zfill(2)}.'
@@ -480,18 +478,24 @@ def download_magnetogram_adapt(time, map_type='fixed', **kwargs):
         raise FileNotFoundError('Could not find a file that matches'
                                 + 'the pattern.')
 
+    directory = kwargs.get('download_dir', './')
+    if directory[-1] != '/':
+        directory += '/'
+
     for filename in filenames:
-        # open the file locally
-        directory = kwargs.get('download_dir', './')
-        if directory[-1] != '/':
-            directory += '/'
-        with open(directory + filename, 'wb') as fhandle:
-            # try to download the magnetogram
-            try:
-                ftp.retrbinary('RETR ' + filename, fhandle.write)
-            except ftplib.all_errors:
-                ftp.quit()
-                raise FileNotFoundError('Cannot download ', filename)
+        # Only try to download if the file does not exist
+        if os.path.isfile(directory+filename) == True:
+            warnings.warn(f'{filename} exists, not downloading',
+                          RuntimeWarning)
+        else:
+            # open the file locally
+            with open(directory + filename, 'wb') as fhandle:
+                # try to download the magnetogram
+                try:
+                    ftp.retrbinary('RETR ' + filename, fhandle.write)
+                except ftplib.all_errors:
+                    ftp.quit()
+                    raise FileNotFoundError('Cannot download ', filename)
 
         # unzip the file
         if '.gz' in filename:

tests/test_swmfpy.py

+#!/usr/bin/env python
+"""Tests for web.py
+"""
+import datetime as dt
+from os.path import isfile
+from swmfpy import *
+
+TIMES = (dt.datetime(2012, 12, 1), dt.datetime(2013, 1, 1))
+
+
+def test_write_imf_from_omni():
+    write_imf_from_omni(*TIMES)
+    assert isfile('IMF.dat')
+

tests/test_tools.py

+#!/usr/bin/env python
+"""Tests for tools.py
+"""
+import numpy as np
+from swmfpy.tools import *
+
+def test_interp_nans():
+    y__ = np.array([1, 1, np.NaN, 1])
+    x__ = np.ones_like(y__)
+    assert not any(np.isnan(interp_nans(x__, y__)))

tests/test_web.py

+#!/usr/bin/env python
+"""Test for web.py
+"""
+from os.path import isfile
+import datetime as dt
+from swmfpy.web import *
+
+TIME = dt.datetime(2016, 2, 3, 2, 1, 1)
+
+
+def test_download_magnetogram_adapt():
+    download_magnetogram_adapt(TIME)
+    assert isfile('adapt40311_03k012_201602030200_i00015600n1.fts')