Transfer Functions

There exists the ability to store transfer functions in an MTH5 for a comprehensive representation of data. The transfer functions are stored at the Station level in a group called TransferFunctionsGroup. Within this group station transfer functions are stored as a single group for each transfer function TransferFunctionGroup. Each TransferFunctionGroup has a data set for each statistical estimate provided within the transfer function. Currently supported estimates are

Estimate

Description

Shape

Data Type

transfer_function

Full transfer function

n_periods x n_inputs x n_outputs

complex

transfer_function_error

Full transfer function error estimation

n_periods x n_inputs x n_outputs

real

impedance

Only horizontal components, traditional impedance tensor

n_periods x 2 x 2

complex

impedance_error

Only horizontal components, traditional impedance tensor error

n_periods x 2 x 2

real

tipper

horizontal and vertical magnetic transfer function, Tipper

n_periods x 1 x 2

complex

tipper_error

horizontal and vertical magnetic transfer function, Tipper error

n_periods x 1 x 2

real

inverse_signal_power

covariance of input channels (sources)

n_periods x n_inputs x n_inputs

complex

residual_covariance

covariance of output channels (responses)

n_periods x n_outputs x n_outputs

complex

Note: There are plans to add phase tensor and resitivity/phase estimations in the future.

This examples demonstrates how transfer functions can be added to an MTH5 file.

TF object

The TF object comes from mt_metadata.transfer_functions.core.TF and is meant to be the common container for transfer functions. It has readers for:

  • EDI

  • EMFT XML

  • Z-files (EMTF output)

  • J-files (BIRRP output)

  • AVG files (Zonge output)

The TF object has two important metadata objects survey_metadata and station_metadata. Metadata from the other files are translated into these containers and translated back when writing a file.

The statistical estimates are stored as xarray.Datasets that have coordinates of period, input_channels, output_channels. This way the transfer function can be generalized. impedance and tipper are stored in transfer_function and TF provides convenience functions to access impedance and tipper and associated errors. Variances are stored as covariances for input channels (inverse_signal_power) and output channels (residual_covariance) when possible, and the transfer_function_error is stored as well.

Note: There are future plans to include phase tensor and resistivity/phase representation as well.

[1]:
from mth5.mth5 import MTH5

from mt_metadata import TF_XML, TF_EDI_SPECTRA, TF_ZMM, TF_EDI_CGG
from mt_metadata.transfer_functions.core import TF
2023-04-21 10:31:36,442 [line 141] mth5.setup_logger - INFO: Logging file can be found C:\Users\jpeacock\OneDrive - DOI\Documents\GitHub\mth5\logs\mth5_debug.log
[2]:
m = MTH5(file_version="0.2.0")
m.open_mth5(r"transfer_function_example.h5", "w")

2023-04-21 10:31:42,346 [line 672] mth5.mth5.MTH5._initialize_file - INFO: Initialized MTH5 0.2.0 file transfer_function_example.h5 in mode w

Read in Transfer functions

  • TF_XML: An example of EMTF XML format, the preferred format for archiving

  • TF_EDI_SPECTRA: An example of an EDI file stored as spectra

  • TF_EDI_CGG: An example of an output file from a contractor

  • TF_ZMM: An example of an output file from EMFT

[3]:
tf1 = TF(TF_XML)
tf1.read_tf_file()

tf2 = TF(TF_EDI_SPECTRA)
tf2.read_tf_file()

tf3 = TF(TF_EDI_CGG)
tf3.read_tf_file()

tf4 = TF(TF_ZMM)
tf4.read_tf_file()

Add TF_XML to the MTH5

When we add a transfer function to an MTH5, it looks for the survey.id and station.id, if it doesn’t find any then they are created. If information is provided on which runs were processed and channels used those are filled in as well.

Note: If you have multiple transfer functions for a given station be sure to rename the file, for an EDI this is in HEADER under the attribute DATAID. Name it something like station_sample_rate or station_runs

[4]:
tf_group_01 = m.add_transfer_function(tf1)
tf_group_01
2023-04-21 10:31:49,566 [line 1054] mth5.mth5.MTH5.get_survey - WARNING: /Experiment/Surveys/CONUS_South does not exist, check survey_list for existing names.
[4]:
/Experiment/Surveys/CONUS_South/Stations/NMX20/Transfer_Functions/NMX20:
====================
    --> Dataset: inverse_signal_power
    ...................................
    --> Dataset: period
    .....................
    --> Dataset: residual_covariance
    ..................................
    --> Dataset: transfer_function
    ................................
    --> Dataset: transfer_function_error
    ......................................
[5]:
est = tf_group_01.get_estimate("transfer_function")
tf_group_01.has_estimate("covariance")
[5]:
True

Have a look at what was added to the MTH5

Note that an EMTF XML has comprehensive metadata which can be used to populate the MTH5 as necessary, including Runs and Channels

Add an example EDI

Here the survey is not specified therefore we need to fill that information in ourselves otherwise an error is raised, see below.

[6]:
#tf_group_02 = m.add_transfer_function(tf2)

Here we give the survey the id unknown_survey. Also note that because the data are stored as spectra in the EDI we can calculate the inverse_signal_power and residual_covariance.

[6]:
tf2.survey_metadata.id = "unknown_survey"
tf_group_02 = m.add_transfer_function(tf2)
print(tf_group_02.has_estimate("covariance"))
2023-04-21 10:32:07,195 [line 1054] mth5.mth5.MTH5.get_survey - WARNING: /Experiment/Surveys/unknown_survey does not exist, check survey_list for existing names.
True

Add typical EDI file

This file only has impedance and tipper and minimal metadata, which are converted into a full transfer function for storage.

[7]:
tf3.survey_metadata.id = "unknown_survey"
tf_group_03 = m.add_transfer_function(tf3)
tf_group_03.has_estimate("covariance")
2023-04-21 10:32:11,616 [line 222] mth5.groups.base.TransferFunction.get_estimate - ERROR: residual_covariance does not exist, check groups_list for existing names
[7]:
False

Add an output from EMTF

A ZMM file contains the full covariance and transfer functions but has minimal metadata

[8]:
print(tf4)
Station: 300
--------------------------------------------------
        Survey:            None
        Project:           None
        Acquired by:       None
        Acquired date:     1980-01-01
        Latitude:          34.727
        Longitude:         -115.735
        Elevation:         0.000
        Declination:
                Value:     13.1
                Model:     WMM
        Coordinate System: geographic
        Impedance:         True
        Tipper:            True
        N Periods:     38
        Period Range:
                Min:   1.16364E+00 s
                Max:   1.09227E+04 s
        Frequency Range:
                Min:   9.15527E-05 Hz
                Max:   8.59372E-01 Hz
[9]:
tf4.survey_metadata.id = "unknown_survey"
tf_group_04 = m.add_transfer_function(tf4)
tf_group_04.has_estimate("impedance")
[9]:
True

Have a look at the MTH5 file

Everything has been filled in now in the MTH5 including metadata about runs and channels

[10]:
m
[10]:
/:
====================
    |- Group: Experiment
    --------------------
        |- Group: Reports
        -----------------
        |- Group: Standards
        -------------------
            --> Dataset: summary
            ......................
        |- Group: Surveys
        -----------------
            |- Group: CONUS_South
            ---------------------
                |- Group: Filters
                -----------------
                    |- Group: coefficient
                    ---------------------
                    |- Group: fap
                    -------------
                    |- Group: fir
                    -------------
                    |- Group: time_delay
                    --------------------
                    |- Group: zpk
                    -------------
                |- Group: Reports
                -----------------
                |- Group: Standards
                -------------------
                    --> Dataset: summary
                    ......................
                |- Group: Stations
                ------------------
                    |- Group: NMX20
                    ---------------
                        |- Group: NMX20a
                        ----------------
                            --> Dataset: ex
                            .................
                            --> Dataset: ey
                            .................
                            --> Dataset: hx
                            .................
                            --> Dataset: hy
                            .................
                            --> Dataset: hz
                            .................
                        |- Group: NMX20b
                        ----------------
                            --> Dataset: ex
                            .................
                            --> Dataset: ey
                            .................
                            --> Dataset: hx
                            .................
                            --> Dataset: hy
                            .................
                            --> Dataset: hz
                            .................
                        |- Group: Transfer_Functions
                        ----------------------------
                            |- Group: NMX20
                            ---------------
                                --> Dataset: inverse_signal_power
                                ...................................
                                --> Dataset: period
                                .....................
                                --> Dataset: residual_covariance
                                ..................................
                                --> Dataset: transfer_function
                                ................................
                                --> Dataset: transfer_function_error
                                ......................................
            |- Group: unknown_survey
            ------------------------
                |- Group: Filters
                -----------------
                    |- Group: coefficient
                    ---------------------
                    |- Group: fap
                    -------------
                    |- Group: fir
                    -------------
                    |- Group: time_delay
                    --------------------
                    |- Group: zpk
                    -------------
                |- Group: Reports
                -----------------
                |- Group: Standards
                -------------------
                    --> Dataset: summary
                    ......................
                |- Group: Stations
                ------------------
                    |- Group: 300
                    -------------
                        |- Group: 300a
                        --------------
                            --> Dataset: ex
                            .................
                            --> Dataset: ey
                            .................
                            --> Dataset: hx
                            .................
                            --> Dataset: hy
                            .................
                            --> Dataset: hz
                            .................
                        |- Group: Transfer_Functions
                        ----------------------------
                            |- Group: 300
                            -------------
                                --> Dataset: inverse_signal_power
                                ...................................
                                --> Dataset: period
                                .....................
                                --> Dataset: residual_covariance
                                ..................................
                                --> Dataset: transfer_function
                                ................................
                                --> Dataset: transfer_function_error
                                ......................................
                    |- Group: SAGE_2005
                    -------------------
                        |- Group: SAGE_2005a
                        --------------------
                            --> Dataset: ex
                            .................
                            --> Dataset: ey
                            .................
                            --> Dataset: hx
                            .................
                            --> Dataset: hy
                            .................
                            --> Dataset: hz
                            .................
                        |- Group: Transfer_Functions
                        ----------------------------
                            |- Group: SAGE_2005
                            -------------------
                                --> Dataset: inverse_signal_power
                                ...................................
                                --> Dataset: period
                                .....................
                                --> Dataset: residual_covariance
                                ..................................
                                --> Dataset: transfer_function
                                ................................
                                --> Dataset: transfer_function_error
                                ......................................
            |- Group: unknown_survey_001
            ----------------------------
                |- Group: Filters
                -----------------
                    |- Group: coefficient
                    ---------------------
                    |- Group: fap
                    -------------
                    |- Group: fir
                    -------------
                    |- Group: time_delay
                    --------------------
                    |- Group: zpk
                    -------------
                |- Group: Reports
                -----------------
                |- Group: Standards
                -------------------
                    --> Dataset: summary
                    ......................
                |- Group: Stations
                ------------------
                    |- Group: TEST01
                    ----------------
                        |- Group: TEST01a
                        -----------------
                            --> Dataset: ex
                            .................
                            --> Dataset: ey
                            .................
                            --> Dataset: hx
                            .................
                            --> Dataset: hy
                            .................
                            --> Dataset: hz
                            .................
                        |- Group: Transfer_Functions
                        ----------------------------
                            |- Group: TEST01
                            ----------------
                                --> Dataset: period
                                .....................
                                --> Dataset: transfer_function
                                ................................
                                --> Dataset: transfer_function_error
                                ......................................
        --> Dataset: channel_summary
        ..............................
        --> Dataset: tf_summary
        .........................

Get a transfer function object from MTH5

To retrieve a transfer function from the MTH5 file a convenience function m.get_transfer_function is supplied. You only need to know the station.id, tf.id, and the survey.id. Here the tf.id is the same as the station.id.

[11]:
tf1_h5 = m.get_transfer_function(tf1.station_metadata.id, tf1.tf_id, tf1.survey_metadata.id)
[12]:
print(tf1)
print(tf1_h5)
Station: NMX20
--------------------------------------------------
        Survey:            CONUS South
        Project:           USMTArray
        Acquired by:       National Geoelectromagnetic Facility
        Acquired date:     2020-09-20
        Latitude:          34.471
        Longitude:         -108.712
        Elevation:         1940.050
        Declination:
                Value:     9.09
                Model:     WMM
        Coordinate System: geographic
        Impedance:         True
        Tipper:            True
        N Periods:     33
        Period Range:
                Min:   4.65455E+00 s
                Max:   2.91271E+04 s
        Frequency Range:
                Min:   3.43323E-05 Hz
                Max:   2.14844E-01 Hz
Station: NMX20
--------------------------------------------------
        Survey:            CONUS South
        Project:           USMTArray
        Acquired by:       National Geoelectromagnetic Facility
        Acquired date:     2020-09-20
        Latitude:          34.471
        Longitude:         -108.712
        Elevation:         1940.050
        Declination:
                Value:     9.09
                Model:     WMM
        Coordinate System: geographic
        Impedance:         True
        Tipper:            True
        N Periods:     33
        Period Range:
                Min:   4.65455E+00 s
                Max:   2.91271E+04 s
        Frequency Range:
                Min:   3.43323E-05 Hz
                Max:   2.14844E-01 Hz

Summarize what transfer functions are in the file

In the MTH5 file there is a property called tf_summary, this provides an array that can be converted to a pandas.DataFrame of the transfer functions within the file. There is a column called hdf5_reference which can be used to get the transfer function directly. This table is updated when the file is close, so when you open the file next it should be up to date. This table should be read only, if you want to change metadata, you should do it directly in the transfer function object.

[13]:
m.tf_summary.clear_table()
m.tf_summary.summarize()
tf_df = m.tf_summary.to_dataframe()
tf_df
[13]:
station survey latitude longitude elevation tf_id units has_impedance has_tipper has_covariance period_min period_max hdf5_reference station_hdf5_reference
0 NMX20 CONUS_South 34.470528 -108.712288 1940.05 NMX20 none True True True 4.654550 29127.110000 <HDF5 object reference> <HDF5 object reference>
1 300 unknown_survey 34.727000 -115.735000 0.00 300 none True True True 1.163640 10922.666990 <HDF5 object reference> <HDF5 object reference>
2 SAGE_2005 unknown_survey 35.550000 -106.283333 0.00 SAGE_2005 none True True True 0.004196 209.731544 <HDF5 object reference> <HDF5 object reference>
3 TEST01 unknown_survey_001 -30.930285 127.229230 175.27 TEST01 none True True False 0.001212 1211.527490 <HDF5 object reference> <HDF5 object reference>

Get TF from reference

[14]:
tf_object = m.from_reference(tf_df.iloc[0].hdf5_reference)

print(tf_object)
Station: NMX20
--------------------------------------------------
        Survey:            CONUS South
        Project:           USMTArray
        Acquired by:       National Geoelectromagnetic Facility
        Acquired date:     2020-09-20
        Latitude:          34.471
        Longitude:         -108.712
        Elevation:         1940.050
        Declination:
                Value:     9.09
                Model:     WMM
        Coordinate System: geographic
        Impedance:         True
        Tipper:            True
        N Periods:     33
        Period Range:
                Min:   4.65455E+00 s
                Max:   2.91271E+04 s
        Frequency Range:
                Min:   3.43323E-05 Hz
                Max:   2.14844E-01 Hz

MTpy

To analyze, plot, prepare input files one should look to use MTpy. Note: MTpy version 2.0 will use MTH5 as the storage mechanism and TF to read/write files.

[15]:
m.close_mth5()
2023-04-21 10:32:58,441 [line 753] mth5.mth5.MTH5.close_mth5 - INFO: Flushing and closing transfer_function_example.h5
[ ]: