Transfer Functions
There exists the ability to store transfer functions in an MTH5 for a comprehensive representation of data. The transfer functions are stored at the Station
level in a group called TransferFunctionsGroup
. Within this group station transfer functions are stored as a single group for each transfer function TransferFunctionGroup
. Each TransferFunctionGroup
has a data set for each statistical estimate provided within the transfer function. Currently supported estimates are
Estimate |
Description |
Shape |
Data Type |
---|---|---|---|
transfer_function |
Full transfer function |
n_periods x n_inputs x n_outputs |
complex |
transfer_function_error |
Full transfer function error estimation |
n_periods x n_inputs x n_outputs |
real |
impedance |
Only horizontal components, traditional impedance tensor |
n_periods x 2 x 2 |
complex |
impedance_error |
Only horizontal components, traditional impedance tensor error |
n_periods x 2 x 2 |
real |
tipper |
horizontal and vertical magnetic transfer function, Tipper |
n_periods x 1 x 2 |
complex |
tipper_error |
horizontal and vertical magnetic transfer function, Tipper error |
n_periods x 1 x 2 |
real |
inverse_signal_power |
covariance of input channels (sources) |
n_periods x n_inputs x n_inputs |
complex |
residual_covariance |
covariance of output channels (responses) |
n_periods x n_outputs x n_outputs |
complex |
Note: There are plans to add phase tensor and resitivity/phase estimations in the future.
This examples demonstrates how transfer functions can be added to an MTH5 file.
TF object
The TF object comes from mt_metadata.transfer_functions.core.TF
and is meant to be the common container for transfer functions. It has readers for:
EDI
EMFT XML
Z-files (EMTF output)
J-files (BIRRP output)
AVG files (Zonge output)
The TF
object has two important metadata objects survey_metadata
and station_metadata
. Metadata from the other files are translated into these containers and translated back when writing a file.
The statistical estimates are stored as xarray.Datasets
that have coordinates of period
, input_channels
, output_channels
. This way the transfer function can be generalized. impedance
and tipper
are stored in transfer_function
and TF
provides convenience functions to access impedance
and tipper
and associated errors. Variances are stored as covariances for input channels (inverse_signal_power
) and output channels (residual_covariance
) when possible,
and the transfer_function_error
is stored as well.
Note: There are future plans to include phase tensor and resistivity/phase representation as well.
[1]:
from mth5.mth5 import MTH5
from mt_metadata import TF_XML, TF_EDI_SPECTRA, TF_ZMM, TF_EDI_CGG
from mt_metadata.transfer_functions.core import TF
2023-04-21 10:31:36,442 [line 141] mth5.setup_logger - INFO: Logging file can be found C:\Users\jpeacock\OneDrive - DOI\Documents\GitHub\mth5\logs\mth5_debug.log
[2]:
m = MTH5(file_version="0.2.0")
m.open_mth5(r"transfer_function_example.h5", "w")
2023-04-21 10:31:42,346 [line 672] mth5.mth5.MTH5._initialize_file - INFO: Initialized MTH5 0.2.0 file transfer_function_example.h5 in mode w
Read in Transfer functions
TF_XML: An example of EMTF XML format, the preferred format for archiving
TF_EDI_SPECTRA: An example of an EDI file stored as spectra
TF_EDI_CGG: An example of an output file from a contractor
TF_ZMM: An example of an output file from EMFT
[3]:
tf1 = TF(TF_XML)
tf1.read_tf_file()
tf2 = TF(TF_EDI_SPECTRA)
tf2.read_tf_file()
tf3 = TF(TF_EDI_CGG)
tf3.read_tf_file()
tf4 = TF(TF_ZMM)
tf4.read_tf_file()
Add TF_XML to the MTH5
When we add a transfer function to an MTH5, it looks for the survey.id
and station.id
, if it doesn’t find any then they are created. If information is provided on which runs were processed and channels used those are filled in as well.
Note: If you have multiple transfer functions for a given station be sure to rename the file, for an EDI this is in HEADER
under the attribute DATAID
. Name it something like station_sample_rate
or station_runs
[4]:
tf_group_01 = m.add_transfer_function(tf1)
tf_group_01
2023-04-21 10:31:49,566 [line 1054] mth5.mth5.MTH5.get_survey - WARNING: /Experiment/Surveys/CONUS_South does not exist, check survey_list for existing names.
[4]:
/Experiment/Surveys/CONUS_South/Stations/NMX20/Transfer_Functions/NMX20:
====================
--> Dataset: inverse_signal_power
...................................
--> Dataset: period
.....................
--> Dataset: residual_covariance
..................................
--> Dataset: transfer_function
................................
--> Dataset: transfer_function_error
......................................
[5]:
est = tf_group_01.get_estimate("transfer_function")
tf_group_01.has_estimate("covariance")
[5]:
True
Have a look at what was added to the MTH5
Note that an EMTF XML has comprehensive metadata which can be used to populate the MTH5 as necessary, including Runs
and Channels
Add an example EDI
Here the survey is not specified therefore we need to fill that information in ourselves otherwise an error is raised, see below.
[6]:
#tf_group_02 = m.add_transfer_function(tf2)
Here we give the survey the id unknown_survey
. Also note that because the data are stored as spectra in the EDI we can calculate the inverse_signal_power
and residual_covariance
.
[6]:
tf2.survey_metadata.id = "unknown_survey"
tf_group_02 = m.add_transfer_function(tf2)
print(tf_group_02.has_estimate("covariance"))
2023-04-21 10:32:07,195 [line 1054] mth5.mth5.MTH5.get_survey - WARNING: /Experiment/Surveys/unknown_survey does not exist, check survey_list for existing names.
True
Add typical EDI file
This file only has impedance and tipper and minimal metadata, which are converted into a full transfer function for storage.
[7]:
tf3.survey_metadata.id = "unknown_survey"
tf_group_03 = m.add_transfer_function(tf3)
tf_group_03.has_estimate("covariance")
2023-04-21 10:32:11,616 [line 222] mth5.groups.base.TransferFunction.get_estimate - ERROR: residual_covariance does not exist, check groups_list for existing names
[7]:
False
Add an output from EMTF
A ZMM file contains the full covariance and transfer functions but has minimal metadata
[8]:
print(tf4)
Station: 300
--------------------------------------------------
Survey: None
Project: None
Acquired by: None
Acquired date: 1980-01-01
Latitude: 34.727
Longitude: -115.735
Elevation: 0.000
Declination:
Value: 13.1
Model: WMM
Coordinate System: geographic
Impedance: True
Tipper: True
N Periods: 38
Period Range:
Min: 1.16364E+00 s
Max: 1.09227E+04 s
Frequency Range:
Min: 9.15527E-05 Hz
Max: 8.59372E-01 Hz
[9]:
tf4.survey_metadata.id = "unknown_survey"
tf_group_04 = m.add_transfer_function(tf4)
tf_group_04.has_estimate("impedance")
[9]:
True
Have a look at the MTH5 file
Everything has been filled in now in the MTH5 including metadata about runs
and channels
[10]:
m
[10]:
/:
====================
|- Group: Experiment
--------------------
|- Group: Reports
-----------------
|- Group: Standards
-------------------
--> Dataset: summary
......................
|- Group: Surveys
-----------------
|- Group: CONUS_South
---------------------
|- Group: Filters
-----------------
|- Group: coefficient
---------------------
|- Group: fap
-------------
|- Group: fir
-------------
|- Group: time_delay
--------------------
|- Group: zpk
-------------
|- Group: Reports
-----------------
|- Group: Standards
-------------------
--> Dataset: summary
......................
|- Group: Stations
------------------
|- Group: NMX20
---------------
|- Group: NMX20a
----------------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
|- Group: NMX20b
----------------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
|- Group: Transfer_Functions
----------------------------
|- Group: NMX20
---------------
--> Dataset: inverse_signal_power
...................................
--> Dataset: period
.....................
--> Dataset: residual_covariance
..................................
--> Dataset: transfer_function
................................
--> Dataset: transfer_function_error
......................................
|- Group: unknown_survey
------------------------
|- Group: Filters
-----------------
|- Group: coefficient
---------------------
|- Group: fap
-------------
|- Group: fir
-------------
|- Group: time_delay
--------------------
|- Group: zpk
-------------
|- Group: Reports
-----------------
|- Group: Standards
-------------------
--> Dataset: summary
......................
|- Group: Stations
------------------
|- Group: 300
-------------
|- Group: 300a
--------------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
|- Group: Transfer_Functions
----------------------------
|- Group: 300
-------------
--> Dataset: inverse_signal_power
...................................
--> Dataset: period
.....................
--> Dataset: residual_covariance
..................................
--> Dataset: transfer_function
................................
--> Dataset: transfer_function_error
......................................
|- Group: SAGE_2005
-------------------
|- Group: SAGE_2005a
--------------------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
|- Group: Transfer_Functions
----------------------------
|- Group: SAGE_2005
-------------------
--> Dataset: inverse_signal_power
...................................
--> Dataset: period
.....................
--> Dataset: residual_covariance
..................................
--> Dataset: transfer_function
................................
--> Dataset: transfer_function_error
......................................
|- Group: unknown_survey_001
----------------------------
|- Group: Filters
-----------------
|- Group: coefficient
---------------------
|- Group: fap
-------------
|- Group: fir
-------------
|- Group: time_delay
--------------------
|- Group: zpk
-------------
|- Group: Reports
-----------------
|- Group: Standards
-------------------
--> Dataset: summary
......................
|- Group: Stations
------------------
|- Group: TEST01
----------------
|- Group: TEST01a
-----------------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
|- Group: Transfer_Functions
----------------------------
|- Group: TEST01
----------------
--> Dataset: period
.....................
--> Dataset: transfer_function
................................
--> Dataset: transfer_function_error
......................................
--> Dataset: channel_summary
..............................
--> Dataset: tf_summary
.........................
Get a transfer function object from MTH5
To retrieve a transfer function from the MTH5 file a convenience function m.get_transfer_function
is supplied. You only need to know the station.id
, tf.id
, and the survey.id
. Here the tf.id
is the same as the station.id
.
[11]:
tf1_h5 = m.get_transfer_function(tf1.station_metadata.id, tf1.tf_id, tf1.survey_metadata.id)
[12]:
print(tf1)
print(tf1_h5)
Station: NMX20
--------------------------------------------------
Survey: CONUS South
Project: USMTArray
Acquired by: National Geoelectromagnetic Facility
Acquired date: 2020-09-20
Latitude: 34.471
Longitude: -108.712
Elevation: 1940.050
Declination:
Value: 9.09
Model: WMM
Coordinate System: geographic
Impedance: True
Tipper: True
N Periods: 33
Period Range:
Min: 4.65455E+00 s
Max: 2.91271E+04 s
Frequency Range:
Min: 3.43323E-05 Hz
Max: 2.14844E-01 Hz
Station: NMX20
--------------------------------------------------
Survey: CONUS South
Project: USMTArray
Acquired by: National Geoelectromagnetic Facility
Acquired date: 2020-09-20
Latitude: 34.471
Longitude: -108.712
Elevation: 1940.050
Declination:
Value: 9.09
Model: WMM
Coordinate System: geographic
Impedance: True
Tipper: True
N Periods: 33
Period Range:
Min: 4.65455E+00 s
Max: 2.91271E+04 s
Frequency Range:
Min: 3.43323E-05 Hz
Max: 2.14844E-01 Hz
Summarize what transfer functions are in the file
In the MTH5 file there is a property called tf_summary
, this provides an array that can be converted to a pandas.DataFrame
of the transfer functions within the file. There is a column called hdf5_reference
which can be used to get the transfer function directly. This table is updated when the file is close, so when you open the file next it should be up to date. This table should be read only, if you want to change metadata, you should do it directly in the transfer function object.
[13]:
m.tf_summary.clear_table()
m.tf_summary.summarize()
tf_df = m.tf_summary.to_dataframe()
tf_df
[13]:
station | survey | latitude | longitude | elevation | tf_id | units | has_impedance | has_tipper | has_covariance | period_min | period_max | hdf5_reference | station_hdf5_reference | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | NMX20 | CONUS_South | 34.470528 | -108.712288 | 1940.05 | NMX20 | none | True | True | True | 4.654550 | 29127.110000 | <HDF5 object reference> | <HDF5 object reference> |
1 | 300 | unknown_survey | 34.727000 | -115.735000 | 0.00 | 300 | none | True | True | True | 1.163640 | 10922.666990 | <HDF5 object reference> | <HDF5 object reference> |
2 | SAGE_2005 | unknown_survey | 35.550000 | -106.283333 | 0.00 | SAGE_2005 | none | True | True | True | 0.004196 | 209.731544 | <HDF5 object reference> | <HDF5 object reference> |
3 | TEST01 | unknown_survey_001 | -30.930285 | 127.229230 | 175.27 | TEST01 | none | True | True | False | 0.001212 | 1211.527490 | <HDF5 object reference> | <HDF5 object reference> |
Get TF from reference
[14]:
tf_object = m.from_reference(tf_df.iloc[0].hdf5_reference)
print(tf_object)
Station: NMX20
--------------------------------------------------
Survey: CONUS South
Project: USMTArray
Acquired by: National Geoelectromagnetic Facility
Acquired date: 2020-09-20
Latitude: 34.471
Longitude: -108.712
Elevation: 1940.050
Declination:
Value: 9.09
Model: WMM
Coordinate System: geographic
Impedance: True
Tipper: True
N Periods: 33
Period Range:
Min: 4.65455E+00 s
Max: 2.91271E+04 s
Frequency Range:
Min: 3.43323E-05 Hz
Max: 2.14844E-01 Hz
MTpy
To analyze, plot, prepare input files one should look to use MTpy
. Note: MTpy version 2.0
will use MTH5
as the storage mechanism and TF
to read/write files.
[15]:
m.close_mth5()
2023-04-21 10:32:58,441 [line 753] mth5.mth5.MTH5.close_mth5 - INFO: Flushing and closing transfer_function_example.h5
[ ]: