`landice/ismip6_forcing/atmosphere` fails with an overflow error

[trhille]

The landice/ismip6_forcing/atmosphere fails with an overflow error (full traceback at the end of this), with the relevant lines being:

  File "/global/cfs/cdirs/m4288/users/trhille/compass/compass/landice/tests/ismip6_forcing/atmosphere/process_smb.py", line 154, in run
    write_netcdf(input_file_combined, combined_file_temp)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

and

OverflowError: Python integer 5670752832 out of bounds for int32

The only int32 variable in the dataset being written is year. I tested with input_file_combined.drop_vars(['year']) before the call to write_netcdf and it failed with the same error, so apparently that is not the issue. So, it is unclear what variable is causing the error.

Traceback (most recent call last):
  File "/global/cfs/cdirs/m4288/users/trhille/compass/compass/run/serial.py", line 322, in _log_and_run_test
    _run_test(test_case, available_resources)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/cfs/cdirs/m4288/users/trhille/compass/compass/run/serial.py", line 419, in _run_test
    _run_step(test_case, step, test_case.new_step_log_file,
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              available_resources)
              ^^^^^^^^^^^^^^^^^^^^
  File "/global/cfs/cdirs/m4288/users/trhille/compass/compass/run/serial.py", line 470, in _run_step
    step.run()
    ~~~~~~~~^^
  File "/global/cfs/cdirs/m4288/users/trhille/compass/compass/landice/tests/ismip6_forcing/atmosphere/process_smb.py", line 154, in run
    write_netcdf(input_file_combined, combined_file_temp)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/mpas_tools/io.py", line 156, in write_netcdf
    ds.to_netcdf(
    ~~~~~~~~~~~~^
        out_filename, encoding=encodingDict, format=format, engine=engine
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/xarray/core/dataset.py", line 2109, in to_netcdf
    return to_netcdf(  # type: ignore[return-value]  # mypy cannot resolve the overloads:(
        self,
    ...<10 lines>...
        auto_complex=auto_complex,
    )
  File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/xarray/backends/api.py", line 2100, in to_netcdf
    store.close()
    ~~~~~~~~~~~^^
  File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/xarray/backends/scipy_.py", line 298, in close
    self._manager.close()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/xarray/backends/file_manager.py", line 234, in close
    file.close()
    ~~~~~~~~~~^^
  File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/scipy/io/_netcdf.py", line 298, in close
    self.flush()
    ~~~~~~~~~~^^
  File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/scipy/io/_netcdf.py", line 411, in flush
    self._write()
    ~~~~~~~~~~~^^
  File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/scipy/io/_netcdf.py", line 423, in _write
    self._write_var_array()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/scipy/io/_netcdf.py", line 471, in _write_var_array
    self._write_var_metadata(name)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/scipy/io/_netcdf.py", line 510, in _write_var_metadata
    self._pack_int(vsize)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/scipy/io/_netcdf.py", line 785, in _pack_int
    self.fp.write(array(value, '>i').tobytes())
                  ~~~~~^^^^^^^^^^^^^
OverflowError: Python integer 5670752832 out of bounds for int32

[xylar]

@trhille, are there int64 variables? MPAS doesn't support that format so they get converted to int32 in write_netcdf. That could be the issue. This is a relatively recent change in write_netcdf() but previously int64 variables were incorrectly getting read in with a different format (I think float) in MPAS and they now produce an error.

[trhille]

@xylar, no it looks like everything else is float32 or float64.

[xylar]

If you change write_netcdf() to just input_file_combined.to_netcdf(), does that work? If so, can you look at the result and see which field might be the culprit?

[trhille]

@xylar, that works! Is there a reason to keep write_netcdf(), or can we just change it to using to_netcdf()?

[xylar]

@trhille, in general, you should prefer write_netcdf() whenever the output will be used by MPAS components because it does a lot of important checks to make sure things are compatible. I would take a look at the results form to_netcdf() and try to find out what was causing trouble with regard to int64 if you can. If MALI actually can read the files, it may not be for a good reason since MPAS can't actually read int64 fields.