Make an MTH5 from ZEN data
This notebook provides an example of how to read in ZEN (.Z3D) files into an MTH5.
[2]:
from mth5.mth5 import MTH5
from mth5.clients import MakeMTH5
from mth5_test_data import get_test_data_path
from mth5.io.zen import Z3DCollection
zen_data_path = get_test_data_path("zen")
Z3D Collection
We will use the Z3DCollection to assemble the .z3d files into a logical order by schedule action or run.
Note: n_samples is an estimate based on file size not the data. To get an accurate number you should read in the full file. Same with start and end. start is based on the schedule start time which is usually 2 seconds earlier than the data start because of instrument buffer while chaning sampling rates. end is based on file size and sample rate.
The Z3DCollection.get_runs() will return a two level ordered dictionary (OrderedDict). The first level is keyed by station ID. These objects are in turn ordered dictionaries by run ID. Therefore you can loop over stations and runs.
[4]:
zc = Z3DCollection(zen_data_path)
runs = zc.get_runs(sample_rates=[4096, 256])
print(f"Found {len(runs)} station with {len(runs[list(runs.keys())[0]])} runs")
Found 1 station with 2 runs
[5]:
runs["100"]["sr4096_0001"]
[5]:
| survey | station | run | start | end | channel_id | component | fn | sample_rate | file_size | n_samples | sequence_number | dipole | coil_number | latitude | longitude | elevation | instrument_id | calibration_fn | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 5 | 100 | sr4096_0001 | 2022-05-17 12:59:57+00:00 | 2022-05-17 13:09:53.351807+00:00 | 5 | ey | C:\Users\peaco\OneDrive\Documents\GitHub\mth5_... | 4096.0 | 9641604 | 2442657 | 1 | 56.0 | None | 40.497573 | -116.821187 | 1457.1 | ZEN_024 | None | |
| 6 | 100 | sr4096_0001 | 2022-05-17 12:59:57+00:00 | 2022-05-17 13:09:53.351562500+00:00 | 3 | hz | C:\Users\peaco\OneDrive\Documents\GitHub\mth5_... | 4096.0 | 9644160 | 2442656 | 1 | 0.0 | 2334 | 40.497573 | -116.821187 | 1457.1 | ZEN_024 | None | |
| 7 | 100 | sr4096_0001 | 2022-05-17 12:59:57+00:00 | 2022-05-17 13:09:53.351318359+00:00 | 2 | hy | C:\Users\peaco\OneDrive\Documents\GitHub\mth5_... | 4096.0 | 9644156 | 2442655 | 1 | 0.0 | 2324 | 40.497573 | -116.821187 | 1457.1 | ZEN_024 | None | |
| 8 | 100 | sr4096_0001 | 2022-05-17 12:59:57+00:00 | 2022-05-17 13:09:53.349854+00:00 | 4 | ex | C:\Users\peaco\OneDrive\Documents\GitHub\mth5_... | 4096.0 | 9641572 | 2442649 | 1 | 55.0 | None | 40.497573 | -116.821187 | 1457.1 | ZEN_024 | None | |
| 9 | 100 | sr4096_0001 | 2022-05-17 12:59:57+00:00 | 2022-05-17 13:09:53.348877+00:00 | 1 | hx | C:\Users\peaco\OneDrive\Documents\GitHub\mth5_... | 4096.0 | 9644628 | 2442645 | 1 | 0.0 | 2314 | 40.497573 | -116.821187 | 1457.1 | ZEN_024 | None |
Build MTH5
Now that we have a logical collection of files, lets load them into an MTH5. We will simply loop of the stations, runs, and channels in the ordered dictionary.
There are a few things that we need to keep track of.
The station metadata pulled directly from the Z3D files can be input into the station metadata, be sure to use the
write_metadatamethod to write the metadata to the MTH5.The Z3D files have the coil response and zen response embedded in the file, so we can put those into the appropriate filter container in MTH5. This is important for calibrating later.
Since this is a MTH5 file version 0.2.0 the filters are in the
survey_groupso add them there.If you want to calibrate the data set calibrate to
True.
[7]:
mth5_path = MakeMTH5.from_zen(
zen_data_path,
calibration_path=None, # "path_to_calibration_file_amtant.cal",
survey_id="common_survey_id",
combine=True,
)
2026-01-04T12:24:25.537097-0800 | INFO | mth5.mth5 | _initialize_file | line: 678 | Initialized MTH5 0.2.0 file c:\Users\peaco\OneDrive\Documents\GitHub\mth5\docs\examples\notebooks\from_zen.h5 in mode w
2026-01-04T12:24:32.141700-0800 | INFO | mth5.timeseries.run_ts | _align_channels | line: 576 | Channels do not have a common end, using latest: 2022-05-17T13:09:51.066650390
2026-01-04T12:24:33.740250-0800 | WARNING | mth5.timeseries.run_ts | validate_metadata | line: 1045 | end time of dataset 2022-05-17T13:09:51.066406250+00:00 does not match metadata end 2022-05-17T13:09:51.066650390+00:00 updating metatdata value to 2022-05-17T13:09:51.066406250+00:00
2026-01-04T12:24:42.597277-0800 | WARNING | mth5.timeseries.run_ts | validate_metadata | line: 1045 | end time of dataset 2022-05-17T13:09:51+00:00 does not match metadata end 2022-05-17T13:09:51.066406250+00:00 updating metatdata value to 2022-05-17T13:09:51+00:00
2026-01-04T12:24:42.838477-0800 | WARNING | mth5.timeseries.scipy_filters | resample_poly | line: 515 | New sample rate is not an even number of original sample rate. The ratio is 255.99999999352485. Use the new dimensions with caution.
2026-01-04T12:24:42.891210-0800 | WARNING | mth5.timeseries.run_ts | validate_metadata | line: 1045 | end time of dataset 2022-05-17T15:54:41+00:00 does not match metadata end 2022-05-17T15:54:41.996094+00:00 updating metatdata value to 2022-05-17T15:54:41+00:00
2026-01-04T12:24:43.010068-0800 | WARNING | mth5.timeseries.run_ts | validate_metadata | line: 1045 | end time of dataset 2022-05-17T15:54:41+00:00 does not match metadata end 2022-05-17T13:09:51.066406250+00:00 updating metatdata value to 2022-05-17T15:54:41+00:00
2026-01-04T12:24:43.309792-0800 | WARNING | mth5.groups.run | from_runts | line: 713 | Channel run.id sr4096_0001 != group run.id sr1_0001. Setting to ch.run_metadata.id to sr1_0001
2026-01-04T12:24:43.634180-0800 | WARNING | mth5.groups.run | from_runts | line: 713 | Channel run.id sr4096_0001 != group run.id sr1_0001. Setting to ch.run_metadata.id to sr1_0001
2026-01-04T12:24:43.985726-0800 | WARNING | mth5.groups.run | from_runts | line: 713 | Channel run.id sr4096_0001 != group run.id sr1_0001. Setting to ch.run_metadata.id to sr1_0001
2026-01-04T12:24:44.332801-0800 | WARNING | mth5.groups.run | from_runts | line: 713 | Channel run.id sr4096_0001 != group run.id sr1_0001. Setting to ch.run_metadata.id to sr1_0001
2026-01-04T12:24:44.734087-0800 | WARNING | mth5.groups.run | from_runts | line: 713 | Channel run.id sr4096_0001 != group run.id sr1_0001. Setting to ch.run_metadata.id to sr1_0001
2026-01-04T12:24:45.458788-0800 | INFO | mth5.mth5 | close_mth5 | line: 772 | Flushing and closing c:\Users\peaco\OneDrive\Documents\GitHub\mth5\docs\examples\notebooks\from_zen.h5
2026-01-04T12:24:45.460082-0800 | INFO | mth5.clients.zen | make_mth5_from_zen | line: 170 | Wrote MTH5 file to: c:\Users\peaco\OneDrive\Documents\GitHub\mth5\docs\examples\notebooks\from_zen.h5
MTH5 Structure
Have a look at the MTH5 structure and make sure it looks correct.
[8]:
with MTH5() as m:
m = m.open_mth5(mth5_path)
print(m)
channel_df = m.channel_summary.to_dataframe()
run_df = m.run_summary
/:
====================
|- Group: Experiment
--------------------
|- Group: Reports
-----------------
|- Group: Standards
-------------------
--> Dataset: summary
......................
|- Group: Surveys
-----------------
|- Group: common_survey_id
--------------------------
|- Group: Filters
-----------------
|- Group: coefficient
---------------------
|- Group: dipole_55.00m
-----------------------
|- Group: dipole_56.00m
-----------------------
|- Group: zen_counts2mv
-----------------------
|- Group: fap
-------------
|- Group: ant4_2314_response
----------------------------
--> Dataset: fap_table
........................
|- Group: ant4_2324_response
----------------------------
--> Dataset: fap_table
........................
|- Group: ant4_2334_response
----------------------------
--> Dataset: fap_table
........................
|- Group: fir
-------------
|- Group: time_delay
--------------------
|- Group: zpk
-------------
|- Group: Reports
-----------------
|- Group: Standards
-------------------
--> Dataset: summary
......................
|- Group: Stations
------------------
|- Group: 100
-------------
|- Group: Features
------------------
|- Group: Fourier_Coefficients
------------------------------
|- Group: Transfer_Functions
----------------------------
|- Group: sr1_0001
------------------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
|- Group: sr256_0002
--------------------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
|- Group: sr4096_0001
---------------------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
--> Dataset: channel_summary
..............................
--> Dataset: fc_summary
.........................
--> Dataset: tf_summary
.........................
2026-01-04T12:25:22.984545-0800 | INFO | mth5.mth5 | close_mth5 | line: 772 | Flushing and closing c:\Users\peaco\OneDrive\Documents\GitHub\mth5\docs\examples\notebooks\from_zen.h5
Channel Summary
Have a look at the channel summary and make sure everything looks good.
[9]:
channel_df
[9]:
| survey | station | run | latitude | longitude | elevation | component | start | end | n_samples | sample_rate | measurement_type | azimuth | tilt | units | has_data | hdf5_reference | run_hdf5_reference | station_hdf5_reference | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | common_survey_id | 100 | sr1_0001 | 40.497576 | -116.821188 | 1456.7 | ex | 2022-05-17 12:59:58+00:00 | 2022-05-17 15:54:41+00:00 | 10484 | 1.0 | electric | 0.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 1 | common_survey_id | 100 | sr1_0001 | 40.497576 | -116.821188 | 1456.7 | ey | 2022-05-17 12:59:58+00:00 | 2022-05-17 15:54:41+00:00 | 10484 | 1.0 | electric | 90.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 2 | common_survey_id | 100 | sr1_0001 | 40.497576 | -116.821188 | 1456.7 | hx | 2022-05-17 12:59:58+00:00 | 2022-05-17 15:54:41+00:00 | 10484 | 1.0 | magnetic | 0.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 3 | common_survey_id | 100 | sr1_0001 | 40.497576 | -116.821188 | 1456.7 | hy | 2022-05-17 12:59:58+00:00 | 2022-05-17 15:54:41+00:00 | 10484 | 1.0 | magnetic | 90.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 4 | common_survey_id | 100 | sr1_0001 | 40.497576 | -116.821188 | 1456.7 | hz | 2022-05-17 12:59:58+00:00 | 2022-05-17 15:54:41+00:00 | 10484 | 1.0 | magnetic | 0.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 5 | common_survey_id | 100 | sr256_0002 | 40.497576 | -116.821188 | 1456.7 | ex | 2022-05-17 13:09:58+00:00 | 2022-05-17 15:54:41.996094+00:00 | 2530304 | 256.0 | electric | 0.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 6 | common_survey_id | 100 | sr256_0002 | 40.497576 | -116.821188 | 1456.7 | ey | 2022-05-17 13:09:58+00:00 | 2022-05-17 15:54:41.996094+00:00 | 2530304 | 256.0 | electric | 90.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 7 | common_survey_id | 100 | sr256_0002 | 40.497576 | -116.821188 | 1456.7 | hx | 2022-05-17 13:09:58+00:00 | 2022-05-17 15:54:41.996094+00:00 | 2530304 | 256.0 | magnetic | 0.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 8 | common_survey_id | 100 | sr256_0002 | 40.497576 | -116.821188 | 1456.7 | hy | 2022-05-17 13:09:58+00:00 | 2022-05-17 15:54:41.996094+00:00 | 2530304 | 256.0 | magnetic | 90.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 9 | common_survey_id | 100 | sr256_0002 | 40.497576 | -116.821188 | 1456.7 | hz | 2022-05-17 13:09:58+00:00 | 2022-05-17 15:54:41.996094+00:00 | 2530304 | 256.0 | magnetic | 0.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 10 | common_survey_id | 100 | sr4096_0001 | 40.497576 | -116.821188 | 1456.7 | ex | 2022-05-17 12:59:58+00:00 | 2022-05-17 13:09:51.064697265+00:00 | 2429194 | 4096.0 | electric | 0.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 11 | common_survey_id | 100 | sr4096_0001 | 40.497576 | -116.821188 | 1456.7 | ey | 2022-05-17 12:59:58+00:00 | 2022-05-17 13:09:51.066650390+00:00 | 2429202 | 4096.0 | electric | 90.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 12 | common_survey_id | 100 | sr4096_0001 | 40.497576 | -116.821188 | 1456.7 | hx | 2022-05-17 12:59:58+00:00 | 2022-05-17 13:09:51.063721+00:00 | 2429190 | 4096.0 | magnetic | 0.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 13 | common_survey_id | 100 | sr4096_0001 | 40.497576 | -116.821188 | 1456.7 | hy | 2022-05-17 12:59:58+00:00 | 2022-05-17 13:09:51.066162109+00:00 | 2429200 | 4096.0 | magnetic | 90.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
| 14 | common_survey_id | 100 | sr4096_0001 | 40.497576 | -116.821188 | 1456.7 | hz | 2022-05-17 12:59:58+00:00 | 2022-05-17 13:09:51.066406250+00:00 | 2429201 | 4096.0 | magnetic | 0.0 | 0.0 | digital counts | True | <HDF5 object reference> | <HDF5 object reference> | <HDF5 object reference> |
Run Summary
Check the runs
[10]:
run_df
[10]:
| channel_scale_factors | duration | end | has_data | input_channels | mth5_path | n_samples | output_channels | run | sample_rate | start | station | survey | run_hdf5_reference | station_hdf5_reference | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | {'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '... | 10483.000000 | 2022-05-17 15:54:41+00:00 | True | [hx, hy] | c:/Users/peaco/OneDrive/Documents/GitHub/mth5/... | 10484 | [ex, ey, hz] | sr1_0001 | 1.0 | 2022-05-17 12:59:58+00:00 | 100 | common_survey_id | <HDF5 object reference> | <HDF5 object reference> |
| 2 | {'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '... | 593.064697 | 2022-05-17 13:09:51.064697265+00:00 | True | [hx, hy] | c:/Users/peaco/OneDrive/Documents/GitHub/mth5/... | 2429202 | [ex, ey, hz] | sr4096_0001 | 4096.0 | 2022-05-17 12:59:58+00:00 | 100 | common_survey_id | <HDF5 object reference> | <HDF5 object reference> |
| 1 | {'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '... | 9883.996094 | 2022-05-17 15:54:41.996094+00:00 | True | [hx, hy] | c:/Users/peaco/OneDrive/Documents/GitHub/mth5/... | 2530304 | [ex, ey, hz] | sr256_0002 | 256.0 | 2022-05-17 13:09:58+00:00 | 100 | common_survey_id | <HDF5 object reference> | <HDF5 object reference> |