- 451
- 130 995
hdf5
United States
เข้าร่วมเมื่อ 17 ต.ค. 2017
HDF5 2.0 - Your questions and HDF5 2.0 Discussion with Director of Engineering Dana Robinson 11/5/24
The HDF Group's director of engineering continued the discussion on HDF5 2.0. You can check out the planned work at github.com/HDFGroup/hdf5/wiki/HDF5-2.0-Planning
Projects for HDF5 2.0 include: semantic versioning, complex number support, and dropping the autotools and a few more rarely used things.
This session was recorded on October 29, 2024.
Join us each Tuesday at 12:20 p.m. Central time (US/Canada) by joining the zoom at us06web.zoom.us/j/98286880081
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Projects for HDF5 2.0 include: semantic versioning, complex number support, and dropping the autotools and a few more rarely used things.
This session was recorded on October 29, 2024.
Join us each Tuesday at 12:20 p.m. Central time (US/Canada) by joining the zoom at us06web.zoom.us/j/98286880081
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
มุมมอง: 48
วีดีโอ
Versioned HDF5 - HDF Clinic with Gerd Heber (Call the Doctor) 10/29/24
มุมมอง 174 ชั่วโมงที่ผ่านมา
The HDF Group's executive director Gerd Heber made himself available for community questions and highlighted another great community project, Versioned HDF5 deshaw.github.io/versioned-hdf5/master/index.html Check out Gerd's notes here: forum.hdfgroup.org/t/hdf-clinic-10-29-2024/12753 This session was recorded on October 29, 2024. Join us each Tuesday at 12:20 p.m. Central time (US/Canada) by jo...
Hyperslab vs point-selection in HPC - Scot Breitenfeld on Call the Doctor 10/22/24
มุมมอง 3414 วันที่ผ่านมา
Scot Breitenfeld of The HDF Group made himself available for user questions and reviewed a recent comparison of hyperslab selection versus point selection in an HPC context. This session was recorded on October 22, 2024. Join us each Tuesday at 12:20 p.m. Central time (US/Canada) by joining the zoom at us06web.zoom.us/j/98286880081 Call the Doctor is a series of weekly, unscripted, live events!...
New info on upcoming HDF5 releases - Dana Robinson on Call the Doctor 10/15
มุมมอง 11714 วันที่ผ่านมา
In this session of Call the Doctor, The HDF Group's Director of Engineer Dana Robinson explained why the next (scheduled for April 2025) release of HDF5 will be HDF5 2.0 and what changes you can expect to see in this next major release. This session was recorded on October 15, 2024. Join us each Tuesday at 12:20 p.m. Central time (US/Canada) by joining the zoom at us06web.zoom.us/j/98286880081 ...
h5pyd updates (hsload and h5acl) - John Readey on Call the Doctor 10/8/24
มุมมอง 9828 วันที่ผ่านมา
The HDF Group's John Readey hosted Call the Doctor. He followed up on his topic last month, which was fancy indexing. You can watch that video here th-cam.com/video/zlhhANdDUY4/w-d-xo.html On the forum, a community user did some benchmarking for that new feature. forum.hdfgroup.org/t/numpystyle-fancy-indexing-of-datasets/12538/16 Next, John talked about recent updates to h5pyd github.com/HDFGro...
Dana Robinson on Call the Doctor 10/1/24 - HDF5 1.16 and other updates
มุมมอง 29หลายเดือนก่อน
The HDF Group's Dana Robinson talked about the next release of HDF5 and why we're debating whether it will be 1.16 or 1.14.6. He also briefly talked about the upcoming HEP - HDF Enhancement Process and how that will help provide transparency and access to the project. This session was recorded on October 1, 2024. Join us each Tuesday at 12:20 p.m. Central time (US/Canada) by joining the zoom at...
Fortran & HPC Expert Scot Breitenfeld on Call the Doctor - September 24, 2024
มุมมอง 24หลายเดือนก่อน
Scot Breitenfeld hosted Call the Doctor on September 24. Scot's our HPC and Fortran expert and shared some HPC news including some testing on Aurora. There was a question and ensuing discussion on determining chunking parameters for using compression on parallel output. This session was recorded on September 24, 2024. Join us each Tuesday at 12:20 p.m. Central time (US/Canada) by joining the zo...
A Python script for dataset statistics for a HDF5 file - Aleksandar Jelenak on Call the Doctor 9/17
มุมมอง 111หลายเดือนก่อน
The HDF Group's Aleksandar Jelenak hosted Call the Doctor on Tuesday, 9/17 and talked about a Python script gist.github.com/ajelenak/9f2a634c47426f35cdb897d6a64052ef he wrote for additional (compared to h5stat) dataset statistics for a given HDF5 file. Aleksandar uses this script to inspect the changes after cloud optimizing a file. This session was recorded on September 17, 2024. Join us each ...
SWMR and FancySelections for h5pyd with HSDS - Call the Doctor 9/10
มุมมอง 37หลายเดือนก่อน
The HDF Group's John Readey used his Call the Doctor session to talk about SWMR (single writer multiple reader) and FancySelections for h5pyd with HSDS. Example notebooks referenced by John during this session: - github.com/HDFGroup/h5pyd/blob/master/examples/notebooks/swmr_example.ipynb - github.com/HDFGroup/h5pyd/blob/master/examples/notebooks/fancy_selection.ipynb This session was recorded o...
Upcoming Releases and some communications notes from The HDF Group - Dana Robinson 9/3/24
มุมมอง 862 หลายเดือนก่อน
The HDF Group's Director of Engineering Dana Robinson provided some general updates. Dana talked about upcoming releases of HDF5, HDF4, and HDFView. Communications Coordinator Lori Cooper announced the availability of the 2024 HDF5 User Group Meeting video recordings and talked briefly about The HDF Group's new support site (still a work in progress) at support.hdfgroup.org. Links mentioned dur...
A look at the latest PureHDF Release - Call the Doctor with Gerd Heber 7-30-24
มุมมอง 462 หลายเดือนก่อน
In addition to answering community questions, we will examine the latest PureHDF release(s). PureHDF v1.0.0 and v2.0.0 were released on June 24 (2024), and there have been subsequent minor releases (see Releases · Apollo3zehn/PureHDF · GitHub github.com/Apollo3zehn/PureHDF/releases). We will examine what has changed since v1.0.0-beta.2 and use this opportunity to update the HDF5 tutorial (GitHu...
HPC and Fortran expert Scot Breitenfeld on Call the Doctor 8/27/24
มุมมอง 242 หลายเดือนก่อน
The HDF Group's Scot Breitenfeld ended up using this session to discuss topics including subfiling, support of multidimensional arrays by various languages, the future of Fortran and more with community members and The HDF Group co-workers. In this session, we mentioned Aleksandar Jelenak's Call the Doctor session last week where he discussed Hidefix, github.com/gauteh/hidefix a Rust package fo...
Crashproofing - Neil Fortner, The HDF Group - 2024 HDF5 User Group Meeting - HUG24
มุมมอง 282 หลายเดือนก่อน
From the 2024 HDF5 User Group Meeting (#HUG24) held August 5-7, 2024 in Chicago, IL. Crashproofing - Neil Fortner, The HDF Group THe HDF Group's Chief HDF5 Software Architect, Neil Fortner shared a proposed feature called Crashproofing. We're actively searching for funding to continue this work to implement this new feature. For more information on this conference including all sessions and sli...
Community Discussion - M. Scot Breitenfeld, The HDF Group
มุมมอง 142 หลายเดือนก่อน
From the 2024 HDF5 User Group Meeting (#HUG24) held August 5-7, 2024 in Chicago, IL. Community Discussion - M. Scot Breitenfeld, The HDF Group Scot hosted a community to discussion to share some of the changes made by The HDF Group: community support improvements with a sustaining engineer of the week, training more staff to handle the release process, improving documentation and performance. A...
HDF5 Performance Tuning - M. Scot Breitenfeld, The HDF Group - 2024 HDF5 User Group Meeting HUG24
มุมมอง 792 หลายเดือนก่อน
From the 2024 HDF5 User Group Meeting (#HUG24) held August 5-7, 2024 in Chicago, IL. HDF5 Performance Tuning - M. Scot Breitenfeld, The HDF Group Scot covered some best practices and methods to improve parallel performance with some examples and test results. The HDF Group's software engineer Scot Breitenfeld For more information on this conference including all sessions and slide decks, visit ...
Analytical Data Platform: Divide & Conquer the Multi-Dimensional Gordian Knot - AirMettle, Inc.
มุมมอง 182 หลายเดือนก่อน
Analytical Data Platform: Divide & Conquer the Multi-Dimensional Gordian Knot - AirMettle, Inc.
DTIO: Unifying I/O for HPC and AI - Keith Bateman, Illinois Institute of Technology - HUG24
มุมมอง 182 หลายเดือนก่อน
DTIO: Unifying I/O for HPC and AI - Keith Bateman, Illinois Institute of Technology - HUG24
HDF5 infrastructure in DUNE - Barnali Chowdhury, Argonne National Laboratory - HUG24
มุมมอง 282 หลายเดือนก่อน
HDF5 infrastructure in DUNE - Barnali Chowdhury, Argonne National Laboratory - HUG24
Optimizing molecular dynamics AI model using HDF5 and DYAD - Dr. Hariharan Devarajan, LLNL - HUG24
มุมมอง 212 หลายเดือนก่อน
Optimizing molecular dynamics AI model using HDF5 and DYAD - Dr. Hariharan Devarajan, LLNL - HUG24
HDF5 Subfiling: A Scalable Approach to Exascale I/O - M. Scot Breitenfeld, The HDF Group - HUG24
มุมมอง 282 หลายเดือนก่อน
HDF5 Subfiling: A Scalable Approach to Exascale I/O - M. Scot Breitenfeld, The HDF Group - HUG24
Semantic Search and Natural Language Query over HDF5 - Chenxu Niu, Texas Tech University - HU
มุมมอง 352 หลายเดือนก่อน
Semantic Search and Natural Language Query over HDF5 - Chenxu Niu, Texas Tech University - HU
Distributed Affix-Based Metadata Search in Self-Describing Data Files - Wei Zhang, Ph.D, LBNL
มุมมอง 312 หลายเดือนก่อน
Distributed Affix-Based Metadata Search in Self-Describing Data Files - Wei Zhang, Ph.D, LBNL
Uncharted Territory - Exploring New Frontiers for HDF5 - Quincey Koziol, NVIDIA - HUG24
มุมมอง 712 หลายเดือนก่อน
Uncharted Territory - Exploring New Frontiers for HDF5 - Quincey Koziol, NVIDIA - HUG24
Upcoming HDF5 feature: our progress on HDF5 multi-threading and more - Mainzer & Pourmal / Lifeboat
มุมมอง 282 หลายเดือนก่อน
Upcoming HDF5 feature: our progress on HDF5 multi-threading and more - Mainzer & Pourmal / Lifeboat
Enlarging Effective DRAM Capacity through Hermes - Luke Logan, Gnosis Research Center
มุมมอง 532 หลายเดือนก่อน
Enlarging Effective DRAM Capacity through Hermes - Luke Logan, Gnosis Research Center
Drishti VOL: The performance profiling and tracing HDF5 VOL connector - Jean Luca Bez, LBNL - HUG24
มุมมอง 1032 หลายเดือนก่อน
Drishti VOL: The performance profiling and tracing HDF5 VOL connector - Jean Luca Bez, LBNL - HUG24
Efficient HDF5 Data Access for Exa-scale Scientific Application - Houjun Tang, Berkeley Lab
มุมมอง 142 หลายเดือนก่อน
Efficient HDF5 Data Access for Exa-scale Scientific Application - Houjun Tang, Berkeley Lab
Hermes: A Heterogeneous-Aware Multi-Tiered Distributed I/O Buffering System - Luke Logan, Gnosis
มุมมอง 282 หลายเดือนก่อน
Hermes: A Heterogeneous-Aware Multi-Tiered Distributed I/O Buffering System - Luke Logan, Gnosis
Potential revision to MPI-IO consistency and its impact on HDF5 - Chen Wang, Lawrence Livermore
มุมมอง 152 หลายเดือนก่อน
Potential revision to MPI-IO consistency and its impact on HDF5 - Chen Wang, Lawrence Livermore
I/O model based on HDF5 - Hua Xu, Gnosis Research Center (IIT) - HUG24
มุมมอง 172 หลายเดือนก่อน
I/O model based on HDF5 - Hua Xu, Gnosis Research Center (IIT) - HUG24
We are good with CMake 3.23.0 or before. We have some software requiring 3.23 so it is widely available
A truly awful presentation 😅
Is anything happening regarding SWMR support for Windows? Is the VFD SWMR project still alive? Will this feature gain Windows support?
Can you please edit the audio of this video and re-post it, it is bot clear at all, so much echo which makes it hard to hear. Thank you.
How exactly do you guys support dynamically resizable datasets? You can't append data to HDF5 without the code getting messy.
The irony of this. Just make sure your format is ACID compliant. That's it. You won't have these problems
Save yourself the hassle. You can't append data reliably in HDF. Workarounds exist, but they fail if there's a power cut, corrupting your data. A format that can't safely handle incremental updates is pointless for dynamic datasets.
What should you use then?
@@IronicHavoc Whatever you need. Sometimes a simple csv does the trick. If you need reliability, relationships and large data just use a normal SQL/NoSQL database. Time series DBs exist. Or you can just use the file system.
@chudchadanstud That's what I assumed you meant. Just asking for the record
@@IronicHavoc I really wish they made it ACID compliant. Most of these issues will be solved. Even as a basic storage solution your entire file will get corrupted if your system crashes while saving a large dataset. It's a massive oversight. Apple wrote a paper once about file formats. They said always assume your system will fail and handle such cases. Never leave it to the end user to deal with it.
@@chudchadanstud I will say I think this stack exchange answer describes the "ideal" use case for H5s fairly well: "HDF is a good complement to databases, it may make sense to run a query to produce a roughly memory-sized dataset and then cache it in HDF if the same data would be used more than once. If you have a dataset which is fixed, and usually processed as a whole, storing it as a collection of appropriately sized HDF files is not a bad option. If you have a dataset which is updated often, staging some of it as HDF files periodically might still be helpful." In my work we use HDF5 a lot, in part for the local caching use case described above, but also because of bureaucratic restrictions, general disorganization, and a prevailing unfamiliarity with the concept of a database among coworkers. I certainly would like to move away from using it as a long term solution for our data storage (which I agree it should not be used as), but I do see the appeal of a self contained dataset that's capable of a bit more than a CSV file.
nice video
Thank you for the interesting talk! Is there a way to show multiple linear signals in one plot?
Thank you for this very useful video!
Thank you for this very useful video!
Thank you for this very useful video!
Thanks , How i solve this error in my hecras model please tell mes something. hdf5-diag error detected in hdf5 (1.8.11) thread 0 #0000: ..\..\ srcH5p
Hi, please feel free to post your question on the community forum at forum.hdfgroup.org or email help@hdfgroup.org for help.
Thank you so mcuh!
Could you do a video going over setting user credentials and api key in the configuration.yml (and/or other) config files, then building the project? Are you able to update these configurations this way without building the project, or do you have to use hsconfig for something like that? Could you also give some examples of the get, put, post, and delete calls with curl commands after the user auth is set? Thanks!
Hi Matt! Thanks so much for your comment. John Readey will go over your questions on his next Call the Doctor session which will be June 11th at 12:20 p.m. Central time (US/Canada). We'd love to have you on the zoom meeting for any followup questions: us06web.zoom.us/j/98286880081 If you need to chat sooner or discuss more please feel free to email help at hdfgroup.org or visit our forum at forum.hdfgroup.org Again, thanks so much for reaching out. Lori
,!!!!
finally building the good compressors in! looking forward to it and paraview. Bitshuffle+zstd is one of the most valuable ones I hope that becomes a builtin. Definitely interested in GPU / overriding a plugin type implementation in some way too. Maybe the client is the side that would do that though to do it targetted.
Direct link to slide deck: www.hdfgroup.org/wp-content/uploads/2023/08/HUG23_FlashX_Compression.pdf
"Promosm"
I've had the most difficult time using hdf5 in C/C++. I always resort to using xml, unfortunately.
Do you have a script that runs through repacking the files appropriately and using h5py in the appropriate ways to set the cache, etc?
I hear XDMF and I think paraview. Using the CPUs hard should have effect on the NVidia drivers ability to keep the GPUs fed with low latency or do other bookkeeping so assuming the CPUs are doing nothing really doesn't work out in practice IMO. Anyway I once did paraview with a few VFD support patches and it worked. It took longer to open but didn't need the copy. So the question is can HDF internally speed up the opening on distributed filesystems where 1024 ranks or something like this have their single private datasets so no one needs to copy for a single file view.
why arent image max and min not saved as attributes?
Question: why HDF5 ? Can't one simply use the filesystem instead ? It provides all the same capabilities -- metadata, heterogeneous data, big data, fast i/o, cross platform, easy sharing (tar, zip, etc) -- and doens't require learning something new (and a tutorial), maintenance HDFView is not working for my h5 file, nor is HDFCompass; Libraries are not well supported in every language; etc. Go figure ... That reminds me of the quote: “The bureaucracy is expanding to meet the needs of the expanding bureaucracy" ... :D
long question, but it only requires one word to answer: speed.
thanks a lot that's very helpful
Incase anyone else runs into an error following what appears to be the easiest and most simplest example of how to use HDF5, be sure to set the "mode" in the call to "h5py.File" to write. In other words, change "f = h5py.File("file5.hdf5")" to "f = h5py.File("file5.hdf5", 'w')".
Is there a new version of Kita Server available on AWS somewhere?
At HEPS, we are using SWMR as well in file writing
yes to simplifying+dramatic speedup of metadata cache. Sounds like that will give you guys so much back bugs wise and performance wise. Definitely question the heft of your c constructs, strongly recommend experimenting with the many many high quality c++ concurrency datastructures even if you recapture it later in c after you figure out what works.
I have joined a new job where I'm required to use the rhdf5 package. However, I have only always learned with CRAN. I can't seem to be able to install these packages despite multiple tries. It seems there are limited resources on rhdf5 to learn from. How do I install it ? how do I work with it ? what is the pattern of code it uses, because it seems very different from CRAN. Can you please share resources for the same ?
Hi Bansi, thanks for the interest in rhdf5. It's always nice to hear people are trying to use it, sorry you're having difficulty. Probably the best place to start for information about the package, including installation instructions, is the package landing page at bioconductor.org/packages/rhdf5/ There is also the introductory vignette for example of how to use the package bioconductor.org/packages/release/bioc/vignettes/rhdf5/inst/doc/rhdf5.html If you have further questions feel free to post them on the Bioconductor support site (support.bioconductor.org/) and you can report any bugs, including missing documentation, as GitHub issues (github.com/grimbough/rhdf5/issues)
extremely quiet audio
Good explanation, but recording volume is extremely low. Maybe you can reupload with proper audio volume
Matlab uses internally hdf5 where images are also stored inside it but I’m unable to extract data from it using h5py module in python if you read it in time please tell me how to read that
Hi, will you please ask this question in our forum at forum.hdfgroup.org or email to help@hdfgroup.org? Either location will get you a better format for discussion to answer your question.
Hi, thank you for the information. You saved me. Best wishes.
Thanks for the clear presentation. It seems to be the ideal format for my task.
thanks for the explanation but is would be better if you slowed down while you are speaking.
You can adjust the playback speed on the video :)
thanks! that's a very clear explanation!
thank you! maybe even it's too clear introduction xD
Nice talk!
Hi, I'm trying to build Parallel HDF5 on MacBook, but mpich my Mac installed was not correct, the mpi I/o for mpich shall be supported by romio, but even if I type --enable-romio for configure of mpich, it did not generate the important library libmpio.a, is there any solution? Thanks a lot!
HI John, please post your question at forum.hdfgroup.org or email help@hdfgroup.org to get the attention of our engineers. Thanks!
Hi is this stable enough for operation ?
@7:40 Is the chunk size implicitly selected?
I can't tell you how frustrating HDF5, h5py and hdfql all are. Ridiculous level of complexity with very poor documentation. I'm trying to read a variable length of compound "records" from hdf5. h5py segfaults - can't handle this. hdfql - how do you even get at this once you have the cursor pointed to the first record ? e.g., how to parse the compound data from the datatype ? I see no doc/example anywhere.
Hi, please visit the Forum at forum.hdfgroup.org and post your questions with a small code example. h5py and hdfql maintainers also frequent the forum and someone should be able to help you.
Terrible audio, unfortunately,
Thank you very much for the help, Gerd!
I tried using h5cpp -- the first talk -- but there are some functions of documentation that are not really implemented and some examples end in segmentation fault. Example h5::gr_t group = fd_var["/path_to_group"]
Could you show a video working step by step with HDF5 in python or R?
lovely!